WO2023127377A1

WO2023127377A1 - Electronic apparatus, control method, and program

Info

Publication number: WO2023127377A1
Application number: PCT/JP2022/044010
Authority: WO
Inventors: 征志中田; 功一朗井上
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2021-12-28
Filing date: 2022-11-29
Publication date: 2023-07-06

Abstract

The present invention provides, for example, an electronic apparatus that improves operability while suppressing power consumption.　According to the present invention, an electronic apparatus has an imaging element that performs imaging in a first mode or a second mode that has lower power consumption than the first mode and an application processor that executes an application that corresponds to a portion of the human body and a prescribed object when the portion of the human body and the prescribed object have been detected by imaging by the imaging element in the second mode.

Description

Electronic device, control method and program

The present disclosure relates to electronic equipment, control methods, and programs.

Since it is desirable for electronic devices to consume as little power as possible, various techniques have been proposed to achieve low power consumption in electronic devices. For example, Patent Literature 1 below describes an electronic device that shifts to high power consumption operation when a reference object such as a face is detected during low power consumption sensing.

JP 2020-145714 A

With the technology described in Patent Document 1, after shifting to high power consumption operation, the user is required to launch an application for camera shooting or information registration, for example. That is, with the technique described in Patent Document 1, it is difficult to achieve both suppression of power consumption and improvement of operability in electronic devices.

One object of the present disclosure is to provide an electronic device, a control method, and a program that can improve operability when executing an application while reducing power consumption, for example.

The present disclosure, for example,
an imaging device that performs imaging in a first mode or in a second mode that consumes less power than the first mode;
an application processor that executes an application corresponding to the part of the human body and the predetermined object when the part of the human body and the predetermined object are detected by imaging in the second mode of the imaging device; be.

The present disclosure, for example,
The imaging device performs imaging in the first mode or in a second mode that consumes less power than the first mode,
A control method in which an application processor executes an application corresponding to a part of the human body and a predetermined object when the part of the human body and the predetermined object are detected by imaging in the second mode of the imaging device. .

The present disclosure, for example,
The imaging device performs imaging in the first mode or in a second mode that consumes less power than the first mode,
The application processor executes an application corresponding to the part of the human body and the predetermined object when the part of the human body and the predetermined object are detected by imaging the image sensor in the second mode. It is a program to be executed by

The present disclosure, for example,
an imaging element that performs imaging in the first mode or in a second mode that consumes less power than the first mode, and detects whether or not it is in a visible state according to the imaging result in the second mode;
and an application processor that executes a predetermined application when the result of detection is a visible state.

The present disclosure, for example,
The imaging device performs imaging in the first mode or in a second mode that consumes less power than the first mode, and detects whether or not it is in a visible state according to the imaging result in the second mode. ,
A control method in which an application processor executes a predetermined application when the result of detection is a visible state.

The present disclosure, for example,
The imaging device performs imaging in the first mode or in a second mode that consumes less power than the first mode, and detects whether or not it is in a visible state according to the imaging result in the second mode. ,
An application processor is a program that causes a computer to execute a control method for executing a predetermined application when the result of detection is the visible state.

It is a figure showing an example of appearance of a smart phone concerning one embodiment. It is a figure which shows the internal structural example of the smart phone which concerns on one Embodiment. It is a figure which shows the structural example of the image pick-up element which concerns on one Embodiment. 4 is a flowchart for explaining an operation example of a smart phone according to one embodiment; FIG. 4 is a diagram referred to when describing an example of an application that is activated when a part of a human body and a predetermined object are detected; FIG. 4 is a diagram referred to when describing an example of an application that is activated when a part of a human body and a predetermined object are detected; FIG. 4 is a diagram referred to when describing an example of an application that is activated when a part of a human body and a predetermined object are detected; FIG. 4 is a diagram referred to when describing an example of an application that is activated when a part of a human body and a predetermined object are detected; FIG. 4 is a diagram referred to when describing an example of an application that is activated when a part of a human body and a predetermined object are detected; It is a figure for demonstrating a modification. It is a figure for demonstrating a modification. It is a figure for demonstrating a modification. It is a figure for demonstrating a modification. It is a figure for demonstrating a modification. It is a figure for demonstrating a modification. It is a figure for demonstrating a modification. It is a figure for demonstrating a modification. It is a figure for demonstrating a modification. It is a figure for demonstrating a modification. It is a figure for demonstrating a modification. It is a figure for demonstrating a modification. It is a figure for demonstrating a modification.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The description will be given in the following order.
<Issues to be considered in this disclosure>
<One embodiment>
<Modification>
The embodiments and the like described below are preferred specific examples of the present disclosure, and the content of the present disclosure is not limited to these embodiments and the like. In addition, the same reference numerals are given to the same or similar configurations, and redundant explanations are omitted as appropriate.

<Issues to be considered in this disclosure>
First, in order to facilitate understanding of the present disclosure, issues to be considered in the present disclosure will be described.

Mobile terminals such as smartphones and smartwatches (clock-type mobile terminals) turn off the power of the display (hereinafter referred to as sleep mode) or input a predetermined password if there is no operation input for a certain period of time. Otherwise, it is common to transition to a state in which operation input cannot be accepted (hereinafter, appropriately referred to as a locked state). Moreover, in recent years, many applications have been distributed via networks such as the Internet, and it has become possible to install many applications on mobile terminals. These applications include applications for making payments, applications for taking pictures, applications for playing games, applications for making purchases, usage history of medicines, exercise history such as walking, pulse, weight, and blood pressure. An application for registering biometric information, such as a map application, can be exemplified.

For example, consider an example where the user launches and executes an application when the mobile device is locked. The user first performs an operation to unlock the mobile terminal. For example, the user enters a password consisting of numbers and letters into the mobile terminal. After the mobile terminal is unlocked, the user selects a desired application from many applications installed on the mobile terminal and activates the application by performing a touch operation on the icon of the application. . As such, the user has to perform many operations in order to start and run one application. Here, it is conceivable to use the technology described in Patent Document 1 to automatically cancel the sleep state or lock state when a reference object such as a face is detected during sensing with low power consumption. However, even if the sleep state and lock state can be automatically released, the operation for selecting and executing the application is still required, and the operability cannot be further improved. Based on the above points, an embodiment of the present disclosure will be described in detail.

<One embodiment>
[Appearance of smartphone]
In this embodiment, a smartphone will be described as an example of an electronic device. Of course, it is also possible to use other mobile terminals such as tablet computers and smart watches as electronic devices.

FIG. 1 is a diagram showing an example of the appearance of a smartphone (smartphone 100) according to this embodiment. Smartphone 100 has housing 11 . A display 12 is provided on one main surface of the housing 11 . A front camera 13</b>A that captures an image of the user of the smartphone 100 or the like is provided, for example, on the upper side of the display 12 . A rear camera 13B is provided on the main surface opposite to the main surface on which the display 12 is provided. A part of a human body and a predetermined object can be imaged by the front camera 13A and the rear camera 13B. A button 14 for turning on/off the power is provided on the side surface of the housing 11 .

[Example of internal configuration of a smartphone]
FIG. 2 is a block diagram showing an internal configuration example of the smartphone 100 according to this embodiment. The smartphone 100 includes a control unit 101, a microphone 102, an audio signal processing unit 103 connected to the microphone 102, an imaging unit 104, a network unit 105, a network signal processing unit 106 connected to the network unit 105, It has a speaker 107 , an audio reproduction unit 108 connected to the speaker 107 , the display 12 described above, a screen display unit 109 connected to the display 12 , a position sensor 110 and a sensor 111 . The audio signal processing section 103 , imaging unit 104 , network signal processing section 106 , audio reproduction section 108 , screen display section 109 , position sensor 110 and sensor 111 are each connected to the control section 101 .

The control unit 101 is composed of a CPU (Central Processing Unit) and the like. The control unit 101 has a ROM (Read Only Memory) in which the program is stored, a RAM (Random Access Memory) used as a work area when the program is executed, and the like (not shown). there is.). The control unit 101 comprehensively controls the operation of the smartphone 100 .

The control unit 101 has an application processor 101A as a functional block. When a part of a human body and a predetermined object are detected by imaging (details will be described later) in the second mode of an image sensor (image sensor 104B described later) of the imaging unit 104, the application processor 101A detects the human body. , and an application (for example, software that performs a specific process) corresponding to a predetermined object. Here, execution of the application includes at least activation of the application. The execution of the application may include processing for displaying an input screen for each application on the display 12 in response to activation, or may include control for automatically performing registration according to the application. It is possible to set as appropriate how much processing after the application is started is included in the execution of the application. Note that the smartphone 100 may store a plurality of applications. In this case, execution of the application may include a process of selecting a predetermined application from a plurality of applications.

The microphone 102 picks up the user's speech and the like. The audio signal processing unit 103 performs known audio signal processing on audio data of sounds picked up via the microphone 102 .

The imaging unit 104 includes, for example, an optical system 104A such as a lens and an imaging element 104B. A CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor can be applied as the imaging element 104B. In this embodiment, the image sensor 104B has a signal processing circuit 104C. For example, it is configured as a one-chip sensor in which the imaging device 104B and the signal processing circuit 104C are stacked.

The network unit 105 includes an antenna and the like. A network signal processing unit 106 performs modulation/demodulation processing, error correction processing, and the like on data transmitted and received via the network unit 105 .

The audio reproduction unit 108 performs processing for reproducing sound from the speaker 107 . The audio reproduction unit 108 performs known audio signal processing such as amplification processing and D/A conversion processing, for example.

As the display 12, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display can be applied. The screen display unit 109 performs known processing for displaying various information on the display 12 . For example, the screen display unit 109 performs processing for displaying a UI corresponding to the application on the display 12 under the control of the application processor 101A. Note that the display 12 may be configured as a touch panel. In this case, the screen display unit 109 also performs detection processing of the operation position associated with the touch operation.

The position sensor 110 is a positioning unit that measures the current position using, for example, a system called GNSS (Global Navigation Satellite System).

The sensor 111 is a general term for sensors other than the image sensor 104B and the position sensor 110. As the sensor 111 , for example, a sensor that detects the movement and state of the smartphone 100 can be used. Specifically, the sensor 111 includes an acceleration sensor, a gyro sensor, an electronic compass, and the like. Sensors 111 may also include sensors that detect the surrounding environment. These sensors include a sensor that measures temperature, a sensor that measures humidity, a sensor that measures atmospheric pressure, a sensor that measures ambient brightness (illuminance), and the like. Further, the sensor 111 may include a sensor that detects biometric information of the user of the smart phone 100 . These sensors include sensors that detect the user's blood pressure, pulse, sweat glands, body temperature, and fingerprints.

[Configuration example of imaging device]
Next, a configuration example of the imaging element 104B according to this embodiment will be described with reference to FIG. The imaging element 104B has a pixel array section 121 in which light receiving elements such as photodiodes are arranged two-dimensionally. The pixel array section 121 includes a horizontal scanning circuit, a vertical scanning circuit, an A/D (Analog to Digital) circuit, etc., which are connected to the light receiving elements (not shown).

In addition, the image sensor 104B includes, as a signal processing circuit 104C, a system controller 131, a power control unit 132, a clock generation circuit 133, a ring oscillator 134 connected to the clock generation circuit 133, a PLL (Phase Locked Loop) 135, a BIAS 136, a timing It has a generator 137 , a sensor I/F (Interface) 138 , a detection section 139 , a camera signal processing section 140 and an output I/F 141 . The system controller 131 , power control section 132 , timing generator 137 , detection section 139 , camera signal processing section 140 and output I/F 141 are connected to each other via a bus 151 .

The system controller 131 has, for example, a microprocessor, and centrally controls the operation of each unit in the imaging device 104B. The system controller 131 has an output I/F 131A. Commands are transmitted and received between the system controller 131 and the control unit 101 (for example, the application processor 101A) via the output I/F 131A. Commands are transmitted and received via a serial interface such as ^I2C .

The power control unit 132 controls power supplied to each unit. Although the details will be described later, the power control unit 132 controls power supplied to each unit according to the operation mode of the imaging element 104B.

The clock generation circuit 133 generates clock signals based on the outputs of the ring oscillator 134 and the PLL 135 . A clock signal generated by the clock generating circuit 133 is supplied to each part of the imaging element 104B, and operations are performed based on the clock signal.

The BIAS 136 generates and supplies a stable reference voltage and reference current to each circuit that controls the pixel array section 121 and processes output signals.

The timing generator 137 generates various timing signals. Various timing signals generated by the timing generator are supplied to the pixel drive circuit of the pixel array section 121 and the sensor I/F 138 .

The sensor I/F 138 is an interface for outputting image data (for example, digitized image data) output from the pixel array unit 121 to a subsequent stage. The sensor I/F 138 operates based on timing signals supplied from the timing generator 137 . A detection unit 139 and a camera signal processing unit 140 are connected to the rear stage of the sensor I/F 138 .

Based on the image data supplied from the sensor I/F 138, the detection unit 139 detects whether or not the image data includes a part of the human body such as a face or a hand, or a predetermined object. The predetermined object may be a part of the human body, an object grasped by a part of the human body, or an object existing at a position separate from the part of the human body. A learning model obtained by performing machine learning such as DNN (Deep Neural Network) may be applied to the detection processing performed by the detection unit 139 . The detection unit 139 notifies the detection result to the system controller 131 via the bus 151 .

The camera signal processing unit 140 performs known image processing on image data supplied from the sensor I/F 138 . Known image processing includes interpolation processing, color correction, defect correction, and the like. Image data subjected to camera signal processing by the camera signal processing unit 140 is supplied to the control unit 101 via the output I/F 141 . An image based on the image data is displayed on the display 12 by operating the screen display unit 109 based on the control of the control unit 101 . As the output I/F 141, for example, MIPI (Mobile Industry Processor Interface) can be applied.

Note that the front camera 13A and the rear camera 13B may each have a configuration related to the imaging unit 104 described above, or may have a configuration in which a part of the configuration is shared.

[Smartphone operation example]
Next, an operation example of the smartphone according to this embodiment will be described. The smartphone 100 always performs a process of detecting a part of the human body and a predetermined object while the power is on. For example, even in a sleep state or a locked state, a process of detecting a part of the human body and a predetermined object is performed. Of course, there may be a period during which a part of the human body and a predetermined object are not detected, such as during operation of the smartphone 100 . Note that the part of the human body to be detected and the predetermined object may be set by the user, or may be preset in the smartphone 100. Data may be downloaded from a server and set in smartphone 100 .

By the way, it is not preferable from the viewpoint of power consumption to always perform the process of detecting a part of the human body and a predetermined object. Therefore, in the present embodiment, processing for detecting a part of the human body and a predetermined object is performed by operating the imaging device 104B with low power consumption. Specifically, in addition to the normal mode (an example of the first mode) for performing normal imaging such as imaging an object, the power consumption of the imaging element 104B is lower than in the normal mode, and the outer shape of the object is reduced. The imaging element 104B is operable in a detectable mode, more specifically, in a detection mode (an example of the second mode) for detecting parts of the human body and predetermined objects. In addition, in this embodiment, the imaging element 104B has a mode in which imaging is performed with lower power consumption than in the detection mode. This mode is a mode that can detect the presence or absence of a moving object (the shape of the moving object is not captured), and this mode is hereinafter referred to as a moving object detection mode. Further, in this embodiment, the imaging device 104B has an imaging mode that consumes more power than the detection mode and can obtain image data that allows recognition of the pattern of the object. This mode is hereinafter referred to as recognition mode. In the recognition mode, for example, object identification is performed by multi-wavelength sensing.

For example, by reducing parameters related to imaging in the normal mode (hereinafter referred to as imaging parameters as appropriate), the power consumption in the detection mode is made smaller than the power consumption in the normal mode. The imaging parameters include parameters related to the amount of image data obtained by imaging and parameters related to driving during imaging. Specific examples of the former include resolution, gradation, color, imaging region (ROI (Region of Interest)), and wavelength resolution of image data. By making the resolution, etc. of image data obtained by imaging in the detection mode smaller than the resolution, etc. of image data obtained by imaging in the normal mode, the power consumption in imaging in the detection mode is lower than the power consumption in imaging in the normal mode. can also be made smaller. Specific examples of the latter include settings related to the number of drive clocks for the imaging device 104B, the frame rate, the operating state/idle state (non-operating state or operating state with low power) of each functional block, and the like. The driving clock and frame rate during imaging in the detection mode are set lower than those during imaging in the normal mode, and the number of functional blocks in the operating state is reduced as much as possible to reduce the power consumption during imaging in the detection mode. can be made smaller than the power consumption in imaging in the normal mode. Similarly, by reducing the imaging parameters, the power consumption in imaging in the moving object detection mode can be made smaller than the power consumption in imaging in the detection mode. The recognition mode is set with larger imaging parameters than the moving object detection mode and the detection mode. Note that the imaging parameters in the normal mode and the imaging parameters in the recognition mode may be the same. However, since the imaging area is optimized in the recognition mode, the smartphone 100 operates with lower power consumption than in the normal mode.

When the imaging device 104B has the configuration shown in FIG. 3, control of each functional block for each mode is as follows, for example.
・Blocks that are supplied with power and operate as usual regardless of the mode: system controller 131, BIAS 136
・Blocks that do not operate because power is not supplied in the normal mode: power control section 132, detection section 139, ring oscillator 134
・Blocks that operate only in normal mode: PLL 135, camera signal processing unit 140, output I/F 141
A block in which power consumption is optimized by adjusting imaging parameters in detection mode, moving object detection mode, and recognition mode: pixel array section 121, clock generation circuit 133, timing generator 137, sensor I/ F138

Note that when the operation mode of the smartphone 100 is a detection mode, a moving body detection mode, or a recognition mode other than the normal mode, the application processor 101A of the control unit 101 is in a non-operating state or a low power consumption operating state (hereinafter , collectively referred to as hibernation as appropriate). Thereby, not only the power consumption of the image sensor 104B but also the power consumption of the control unit 101 can be reduced.

A specific operation example of the smartphone 100 will be described with reference to the flowchart of FIG.

In step ST11, control is performed to set the operation mode of the smartphone 100 to the moving object detection mode. For example, imaging in the moving object detection mode is started when the smartphone 100 is in a sleep state or locked state, or when the position information is within a predetermined range. For example, when the above-described trigger occurs, the control unit 101 notifies the system controller 131 of the imaging device 104B of that fact. The system controller 131 that has received the notification from the control unit 101 activates the functional blocks necessary for imaging in the moving object detection mode, and deactivates the functional blocks that are unnecessary for imaging in the moving object detection mode.

Specifically, the system controller 131 controls the power control section 132, the ring oscillator 134, and the detection section 139 to operate. The system controller 131 also controls the PLL 135, the camera signal processing unit 140, and the output I/F 141 to be in a rest state. The system controller 131 controls the driving circuit of the pixel array unit 121, the clock generation circuit 133, the timing generator 137, the sensor, and the like, using imaging parameters (for example, resolution, gradation, driving clock, frame rate, etc.) corresponding to the moving object detection mode. An imaging parameter is set for each unit so that the I/F 138 operates. The parameter corresponding to the moving body detection mode is, for example, a parameter set in advance to the extent that the presence or absence of a moving body can be detected. For example, the presence or absence of a moving object can be detected from changes in pixel output, so the number of drive pixels may theoretically be one. Power consumption in functional blocks operating with parameters corresponding to the operation detection mode may be smaller than power consumption during operation in the detection mode or the normal mode. Control for reducing the power supply is performed by the power control section 132 . Then, the process proceeds to step ST12.

At step ST12, it is determined whether or not a moving object has been detected as a result of imaging in the moving object detection mode. For example, the detection unit 139 determines whether or not a moving object is detected based on image data output from the sensor I/F 138 . For example, the detection unit 139 determines that there is a moving object when the pixel value of the pixel output at each predetermined timing changes by a certain amount or more, and determines that there is no moving object when the pixel value does not change by a certain amount or more. to decide. The detector 139 notifies the system controller 131 of the determination result. If there is no moving object (No), the process returns to step ST11, and imaging in the moving object detection mode is repeated. If there is a moving object (Yes), the process proceeds to step ST13.

In step ST13, imaging is performed in the detection mode. For example, the system controller 131 notified of the determination result that there is a moving object from the detection unit 139 uses the imaging parameters corresponding to the detection mode to drive the pixel array unit 121, the clock generation circuit 133, the timing generator 137, and the sensor interface. The imaging parameters are set for each unit so that F138 operates. The parameter corresponding to the detection mode is, for example, a parameter set in advance to the extent that a part of the human body and a predetermined object can be detected. Since it is necessary to improve the recognition accuracy in the detection mode more than in the moving body detection mode, it is necessary to increase the power supplied to the functional block compared to the detection mode. Control for increasing power supply is performed by the power control unit 132 . Then, the process proceeds to step ST14.

In step ST14, it is determined whether or not a part of the human body and a predetermined object are detected as a result of imaging in the detection mode. For example, based on the image data output from the sensor I/F 138, the detection unit 139 determines whether a part of the human body and a predetermined object have been detected. A known process can be applied as the detection process. For example, as processing for detecting a part of the human body, processing for detecting skin color, processing for detecting feature points such as joints, and the like can be applied. As the process for detecting a predetermined object, a process using a learning model obtained by machine learning, a process such as contour detection, etc. can be applied. The detector 139 notifies the system controller 131 of the determination result. Note that when the content of the notification indicates that a part of the human body and a predetermined object have been detected, the system controller 131 transmits an interrupt to the control unit 101 . The control unit 101 that has received the interrupt changes the operation state of the application processor 101A from the sleep state to the operation state. If the part of the human body and the predetermined object are not detected (No), the process returns to step ST13, and the imaging in the detection mode is repeated. If the part of the human body and the predetermined object are detected (Yes), the process proceeds to step ST15.

In step ST15, imaging is performed in the recognition mode. The detection unit 139 supplies image data obtained by imaging in the recognition mode to the system controller 131 . The system controller 131 recognizes a part of the human body or a predetermined object, obtains detailed information about the part of the human body or the predetermined object, and recognizes an application to be activated based on this information. A recognition result is supplied to the application processor 101A. Note that, in imaging in the recognition mode, for example, only a partial area of the human body or a predetermined object area detected in the detection mode is imaged with high image quality (for example, at high resolution). As a result, it is possible to suppress power consumption in imaging in the recognition mode. Then, the process proceeds to step ST16.

In step ST16, the application processor 101A, which has entered an operating state, launches an application corresponding to the recognition result supplied from the system controller 131. Then, the process proceeds to step ST17.

At step ST17, the application started by the application processor 101A is executed. As described above, it is possible to appropriately change up to which stage of the application (the input stage and the stage up to registration) to be executed after the application is started. However, it is preferable that the execution result of the application be notified so that the user can recognize it. For example, it is preferable that the execution result of the application be displayed on the display 12 or be notified by voice through the speaker 107 .

It should be noted that the moving object detection mode, detection mode, and recognition mode may use the front camera 13A or the rear camera 13B in some cases.

[Specific example of application]
(first example)
Next, a specific example of a human body part and a predetermined object, and a specific example of an application activated according to the detection result of the human body part and the predetermined object will be described. Note that the specific examples described below are merely examples, and other examples may exist, and the electronic device according to the present disclosure may correspond to some or all of the following specific examples. Further, the combination of the detected part of the human body and the predetermined object and the application activated according to the detection result may be preset in the smartphone 100, or may be set by the user. Alternatively, the smart phone 100 may be automatically set by performing machine learning.

A first example is an example in which the part of the human body is the user's hand, the predetermined object is an object held or gripped by the user's hand, and the application is a camera. In this example, the application processor 101A recognizes a part of the human body as an instruction to operate an object (for example, an instruction to capture an image), and executes control for capturing an image with the camera according to the recognition result.

For example, as shown in FIG. 5A, image data including the user's

hands

21A and 21B and the document 22 is obtained as a result of imaging in the detection mode. Further, as a result of imaging in the recognition mode, it is detected that the

hands

21A and 21B are holding the document 22. FIG.

The detection unit 139 notifies the system controller 131 that a part of the human body and a predetermined object have been detected. The detection unit 139 also outputs image data obtained in the recognition mode to the system controller 131 . The system controller 131 recognizes the contents of operations corresponding to characters and regions of a document and gestures by performing recognition processing on image data. In this example, since both hands (

hands

21A and 21B) are detected and an object is present between both hands, the system controller 131 determines that the camera is the application to be started, and that the camera is positioned between both hands. It is recognized as an operation instruction to image an object.

The system controller 131 notifies the application processor 101A of the recognition result. The application processor 101A that has received the notification controls the imaging unit 104 to perform imaging. The imaging unit 104 operates according to the control, and the imaging unit 104 performs imaging. Specifically, the area where the document 22 exists is imaged by the imaging device 104B. Note that the document 22 is considered to exist in the ROI, so it is preferable that the image of the document 22 is captured with high image quality. Therefore, the document 22 is imaged, for example, in the normal mode. The image data of the document 22 obtained by imaging in the normal mode is subjected to image processing for improving image quality by the camera signal processing unit 140, and the image data subjected to image processing (see FIG. 5C) is output I /F141 to the control unit 101. Image data of the document 22 is displayed on the display 12 or stored in memory.

(Second example)
A second example is an example in which the part of the human body is the user's finger and the predetermined object is an object positioned at the fingertip. The object may be grasped with fingertips. The application launched in this case is an application that registers information related to the object located at the fingertip. The information related to the object positioned at the fingertip is, for example, information related to the usage history of the object.

For example, as shown in FIG. 6A, as a result of imaging in the detection mode, an image including the user's hand 23 including the thumb 23A and the index finger 23B, and the medicine tablet 24 positioned at the fingertips of the thumb 23A and the index finger 23B. Suppose we have data. The detection unit 139 outputs to the system controller 131 that the part of the human body and the predetermined object have been detected. Also, image data obtained in the recognition mode is supplied to the system controller 131 . By performing recognition processing on image data, the system controller 131 recognizes the positional information of the fingertips of the thumb 23A and the index finger 23B, the fact that the object at the fingertips is the drug tablet 24, the type of the drug, and the like. Furthermore, it recognizes the content of the operation corresponding to the gesture. In this example, since the medicine tablet 24 is present at the fingertip, the system controller 131 recognizes that the application to be started is an application for registering the use history of the medicine tablet 24 . The system controller 131 notifies the recognition result to the application processor 101A.

Upon receiving the notification, the application processor 101A activates an application for registering the medicine usage history. Then, the medicine taking history (for example, the type of medicine taken and the date and time) is registered in the application. FIG. 6B shows an example of a drug registration screen in the application. A screen shown in FIG. 6B is displayed on the display 12 . According to this example, it is possible to easily register the taking history of medicines that tend to be forgotten.

The object located at the fingertip may be the date on the calendar. In this case, the information related to the date on the calendar can include the schedule entered in the column for that date. For example, when the fingertip and the calendar positioned at the fingertip are detected, the schedule function of the smartphone 100 is activated as an application, and the schedule, which is information related to the date of the calendar, is registered in the schedule function.

Also, the part of the human body may be, for example, a face instead of a finger, and the application may be an application that registers the use history of cosmetics instead of the history of taking medicine. For example, it is assumed that the user's face 25 and the container 26 are detected by imaging in the detection mode, as shown in FIG. Further, the imaging in the recognition mode allows the system controller 131 to recognize that the container 26 is a container for lotion. In this case, the system controller 131 recognizes that the application to be activated is an application for registering the use history of cosmetics, for example. The system controller 131 notifies the recognition result to the application processor 101A.

Upon receiving the notification, the application processor 101A activates an application for registering the use history of cosmetics. Then, the use history of cosmetics (for example, the type of cosmetics used and the date and time of use) is registered in the application. In this way, the combination of the part of the human body and the predetermined object can be changed as appropriate, and the application activated according to the combination can also be changed as appropriate.

(Third example)
In a third example, a part of the human body and a part of the human body different from the part of the human body (predetermined object in this example) are detected by imaging in the detection mode, and each part of the human body is detected. This is an example in which an application corresponding to the combination is executed. In this example, the human body part is the fingers of one hand, and the human body part different from the human body part is the fingers of the other hand. An example of an alarm is an application that is activated according to the combination of fingers on each hand.

For example, as shown in FIG. 8A, it is assumed that image data including fingers 31 of one hand and fingers 32 of the other hand are obtained as a result of imaging in the detection mode. The detection unit 139 notifies the system controller 131 that the fingers of each hand have been detected. Also, image data obtained by imaging in the recognition mode is supplied to the system controller 131 . The system controller 131 recognizes the positions of the

fingers

31 and 32 and the number of fingers by performing recognition processing on the image data. In this example, since the

fingers

31 and 32 are present in the image data so as to overlap each other, the system controller 131 recognizes that the application to be activated is, for example, an alarm. The system controller 131 notifies the recognition result to the application processor 101A.

Upon receiving the notification, the application processor 101A activates an application that sets an alarm. Then, the application processor 101A sets an alarm at a time (7 o'clock in this example) corresponding to the total number of fingers 31 and 32 (7 in this example). FIG. 8B shows an example of an alarm setting screen in the application. The screen shown in FIG. 8B (the screen in which the alarm is automatically set at 7:00) is displayed on the display 12 . According to this example, an alarm can be easily set.

Note that the application in this example is not limited to alarms. For example, the application in this example may be the settings of the smart phone 100 . Smartphone 100 has a mode corresponding to private use of smartphone 100 (hereinafter referred to as private mode as appropriate) and a mode corresponding to work use of smartphone 100 (hereinafter referred to as office mode as appropriate). can be set. In this case, the application corresponding to the finger combination may be the control for setting either private mode or office mode. The private mode and the office mode differ, for example, in the destination communication carrier (line operator), the type of application for which various notifications are made, and the arrangement of icons on the display 12 .

For example, as shown in FIGS. 9A and 9B, image data including fingers 31 of one hand and fingers 32 of the other hand is obtained as a result of imaging in the detection mode. The detection unit 139 notifies the system controller 131 that the

fingers

31 and 32 have been detected. Also, image data obtained by imaging in the recognition mode is supplied to the system controller 131 . The system controller 131 recognizes the positions and number of the

fingers

31 and 32 by performing recognition processing on the image data, and further recognizes that the application to be activated is mode setting control. The system controller 131 notifies the recognition result to the application processor 101A.

The application processor 101A that received the notification launches the application for setting the mode. When the total number of fingers 31 and fingers 32 is six as shown in FIG. 9A, the application processor 101A makes settings corresponding to the private mode. Also, when the total number of

fingers

31 and 32 is seven as shown in FIG. 9B, the application processor 101A performs settings corresponding to the office mode. Of course, the number of fingers corresponding to each mode is not limited to six or seven.

[Effect obtained by this embodiment]
According to the present embodiment described above, when a part of the human body and a predetermined object are detected by imaging with low power consumption, an application corresponding to the detected part of the human body and the predetermined object is executed. can be started automatically. The user does not need to unlock the smartphone or select the desired application from a large number of applications, so the operations required to start the application can be greatly reduced. That is, it is possible to improve operability when executing an application while suppressing power consumption.
Since imaging in the detection mode is performed with low power consumption, even when imaging in the detection mode is performed for a long period of time, the remaining battery capacity of the smartphone does not decrease significantly. Since imaging in the detection mode can be performed for a long time, various actions (a combination of a part of the human body and a predetermined object) can be detected, and an application corresponding to the detection result can be activated. That is, it is possible to easily start not only applications that are likely to be executed in a specific time period, but also applications that can be executed at various timings and locations (for example, applications for imaging, payment, scheduling, etc.).

<Modification>
Although the embodiments of the present disclosure have been specifically described above, the content of the present disclosure is not limited to the above-described embodiments, and various modifications are possible based on the technical ideas of the present disclosure. In addition, the same reference numerals are given to the same or similar configurations as those of one embodiment, and redundant explanations are appropriately omitted.

[Modification 1]
First, modification 1 will be described. In this modification and Modifications 2 to 7 described below, the imaging element 104B can perform imaging in the normal mode and in a mode with lower power consumption than the normal mode, as in the first embodiment. The imaging mode with low power consumption means that power is supplied to an appropriate functional block of the signal processing circuit 104C at least within a range in which it is possible to detect whether or not the electronic device is being viewed by the user. , which is a mode for suppressing power consumption. The signal processing circuit 104C to which power corresponding to this mode is supplied performs processing for detecting whether or not the user is in the viewing state. The imaging mode in which the user can detect whether or not the electronic device is in the visible state and consumes less power than the normal mode is hereinafter referred to as the visible state detection mode as appropriate. Although imaging in the visual recognition state detection mode consumes less power than imaging in the normal mode, there are restrictions on power supply. Therefore, the image quality obtained by imaging in the visual recognition state detection mode is lower than the image quality obtained by imaging in the normal mode. Note that the visible state means a state in which the user is viewing an electronic device such as a smartphone. Also, the user in the visible state is appropriately referred to as the visible state person. A specific example of the user is a person, but may be an animal such as a robot or a pet. The application processor 101A executes a predetermined application when the result of detection by the imaging element 104B is the visible state. Specific contents of the modified example will be described below.

10A to 10C are diagrams for schematically explaining the contents of this modification. FIG. 10A shows a state in which user U of smartphone 100 is viewing a map displayed on display 12 . In this state, the smartphone 100 side uses the front camera 13A to determine whether or not the user U is looking at the smartphone 100, that is, whether or not it is in a visible state.

When it is detected that the user U is in the visible state, the application processor 101A performs obstacle detection processing as an example of a predetermined application. Specifically, as shown in FIG. 10B, an image is captured using the rear camera 13B. For example, when the rear camera 13B is controlled to be off, the rear camera 13B is activated. Then, after the rear camera 13B is set to perform imaging in the normal mode, imaging by the rear camera 13B is started. Since the rear camera 13B takes an image in the normal mode, the image quality of the image obtained by the image pick-up is higher than that of the image obtained by the image pick-up in the visual recognition state detection mode. Obstacle detection processing for detecting obstacles is performed using this image. Obstacles include, for example, people, mobile objects such as cars and bicycles, objects in front of the user U, and the like. Since the obstacle detection process is performed using the high-quality image, it is possible to perform the obstacle detection process with high accuracy. When an obstacle in front of the user U is detected as a result of the obstacle detection process, the user U is notified that there is an obstacle ahead by sounding an alarm or vibrating the smartphone 100 .

Assume that the user U stops looking at the smartphone 100 as shown in FIG. 10C. In this case, the non-visible state is detected. Since it is not in the visible state, the imaging by the rear camera 13B ends. Note that, for example, while the smartphone 100 is controlled to be powered on, the front camera 13A continues to capture images in the visual recognition state detection mode.

FIG. 11 is a flowchart showing the flow of processing performed in this modified example. In step ST31, an image is captured in the visual recognition state detection mode using the front camera 13A. In the viewing state detection mode, power is supplied to the pixel array unit 121, the system controller 131, the sensor I/F 138, and the detection unit 139, for example. Then, the process proceeds to step ST32.

In step ST32, viewing state detection processing is performed to determine whether or not the user U of the smartphone 100 is in the viewing state based on the image captured in the viewing state detection mode. For example, if the image captured in the visible state detection mode includes eyes, it is determined to be in the visible state, and if the image captured in the visible state detection mode does not include eyes, the state is not in the visible state ( hereinafter referred to as a non-visible state as appropriate). By detecting not only the presence or absence of the eye region but also the direction of the line of sight, it is possible to more accurately determine whether the state is the visible state or the non-visible state. For example, when the line of sight of the eye included in the image is directed toward the front camera 13A, it is determined to be in the visible state, and when the line of sight of the eye included in the image is not directed to the front camera 13A, it is determined to be in the non-visible state. may It should be noted that whether or not the object is in the visible state may be determined by applying a learning model learned in advance by machine learning. For example, determination of the visible state or the non-visible state may be performed by applying a learning model obtained by learning the eye shape in the visible state. The viewing state detection process is performed by the detection unit 139 in this modified example, but may be performed by another functional block such as the system controller 131 . Then, the process proceeds to step ST33.

At step ST33, it is determined whether or not the degree of certainty of the visibility state is equal to or greater than a threshold. Such determination is made by the detection unit 139, for example. For example, if the visible state lasts for several seconds or if the eye area reflected in the image is large, the confidence level is high, and if the visible state lasts for a short time or the eye area reflected in the image is small, the confidence level is high. become smaller. Then, the detection unit 139 notifies the system controller 131 of the determination result. The system controller 131 that has received the determination result notifies the application processor 101A that the user U is in the visible state when the certainty of the visible state is equal to or greater than the threshold. Further, when the certainty of the visible state is less than the threshold, the system controller 131 notifies the application processor 101A that the user U is in the non-visible state. The processing up to this point (the processing enclosed by the dotted line in FIG. 11) is performed by the image sensor 104B. If the user U is in the visible state (Yes), the process proceeds to step ST35.

The processing after step ST35 is the obstacle detection processing (application for obstacle detection) performed by the application processor 101A. In step ST35, the application processor 101A turns on the display 12 because the user U is in the viewing state. Then, the process proceeds to step ST36.

At step ST36, the application processor 101A performs control to activate the acceleration sensor that constitutes the sensor 111 in this modified example. Then, the process proceeds to step ST37.

At step ST37, the application processor 101A detects whether or not the user U is in a walking state based on the sensing result of the acceleration sensor. If the user U is walking, the process proceeds to step ST38.

In step ST38, after the rear camera 13B is activated, the normal mode of the rear camera 13B starts to shoot. An image captured by the rear camera 13B in normal mode is supplied to the camera signal processing section 140 . The camera signal processing unit 140 performs processing for detecting whether or not an obstacle is included in the image. For example, the camera signal processing unit 140 performs processing for detecting the presence or absence of an obstacle on the road included in the image. Through such processing, monitoring regarding the presence or absence of an obstacle using the rear camera 13B is started. Although not shown, for example, when an obstacle is detected by the camera signal processing unit 140, the application processor 101A is notified to that effect. The application processor 101A that has received the notification notifies the user U of the presence of the obstacle using sound, display, vibration, a combination thereof, or the like.

On the other hand, if the determination result of step ST33 is the non-visible state (No), the process proceeds to step ST39. In step ST39, the application processor 101A turns off the display 12 because the user U is in the non-visible state. If the display 12 is in the off state, the off state is continued. Then, the process proceeds to step ST40.

In step ST40, since the user U is in the non-visible state and there is no need to perform obstacle detection processing, the application processor 101A performs control to stop the operation of the acceleration sensor. When the operation of the acceleration sensor is stopped, the application processor 101A continues the stopped state of the acceleration sensor. Then, the process proceeds to step ST41.

In step ST41, since there is no need to perform obstacle detection processing, imaging by the rear camera 13B ends, and monitoring regarding the presence or absence of obstacles ends. If the determination result in step ST37 is No, that is, if the vehicle is not walking, there is no need to perform the obstacle detection process, so the process proceeds to step ST41 and the imaging by the rear camera 13B ends.

In the above-described processing, the display control for the display 12 and the control for the acceleration sensor (for example, steps ST35 and ST36) may be switched in order or may be performed in parallel. Also, the process of detecting an obstacle based on an image obtained by imaging in the normal mode may be performed by the application processor 101A instead of the camera signal processing section 140. FIG.

Also, the content displayed on the display 12 is not limited to the map, and may be an image of the other party. This modification can be applied even when the user U walks while operating the smartphone 100 related to urgent contact.

According to this modified example, the visual recognition state detection process is performed based on the imaging result with low power consumption. Therefore, for example, power consumption in the smartphone 100 can be suppressed even if the visual recognition state detection process is performed all the time or for a long time after the power of the smartphone 100 is turned on. Further, since the visual recognition state detection processing is performed within the image sensor 104B, the amount of data transferred between the image sensor 104B and the application processor 101A can be reduced, and power consumption can be reduced. In addition, since the application processor 101A executes a predetermined application when it is detected that the user U is in the visible state, unnecessary execution of the application can be prevented. In addition, since it is not necessary to add a sensor other than the image sensor 104B to perform the visual recognition state detection process, it is possible to reduce the number of parts and the manufacturing cost. In addition, since the application for detecting obstacles is activated and executed without the user U performing an operation, convenience for the user U can be improved.

[Modification 2]
FIG. 12 is a diagram showing an appearance example of an imaging device (imaging device 100A) according to this modification. The imaging device 100A has, for example, a substantially rectangular parallelepiped shape. The imaging device 100A also has a display 12A on one main surface. Furthermore, the imaging device 100A has a first camera 13C provided on the same surface as the display 12A, and a second camera 13D provided on a side perpendicular to the display 12A. The first camera 13C and the second camera 13D have an imaging device 104B. The imaging device 100A has a configuration that can be attached to a moving object such as a car, a bicycle, or a motorcycle using an appropriate attachment member. The imaging device 100A has, for example, functions similar to those of the smartphone 100, but may have functional differences.

13A to 13C are diagrams for schematically explaining the contents of this modification. The examples shown in FIGS. 13A to 13C are examples in which the imaging device 100A is attached to a bicycle. For example, when the power is turned on, the imaging device 100A starts imaging in the visual recognition state detection mode using the first camera 13C.

As shown in FIG. 13A, it is assumed that the user U looks at the display 12A while the bicycle is stopped and confirms the route of the bicycle. It is detected based on the image obtained by the first camera 13C that the user U is in the visible state.

Then, as shown in FIG. 13B, it is assumed that the user U starts the bicycle while confirming the route (while looking at the display 12A). Since the user U is in the visible state and the user U has moved, after the second camera 13D is activated, the second camera 13D takes an image in the normal mode. The above-described obstacle detection processing is performed based on the image obtained by the second camera 13D. When an obstacle is detected in front of the user U, the user U is notified of this fact. During this time as well, imaging in the visual recognition state detection mode using the first camera 13C is continued.

Then, as shown in FIG. 13C, when the user U begins to ride the bicycle while looking ahead, it is detected that the user U is in the non-visible state based on the image obtained by imaging in the visible state detection mode. be. In this case, the imaging device 100A determines that the obstacle detection process does not need to be performed because it is considered that the user U is facing the front without looking at the smartphone 100 . Based on such determination, the imaging by the second camera 13D ends. In addition, since it is assumed that the user U will look at the display 12A again to confirm the route of the bicycle, the first camera 13C captures an image in the visual recognition state detection mode and the visual recognition state detection process using the imaging result. continues to take place.

FIG. 14 is a flowchart showing the flow of processing according to this modified example. Since the contents of steps ST41 to ST43 are the same as the processes of steps ST31 to ST33 in Modification 1, except that the front camera 13A is the first camera 13C, redundant description will be omitted.

The process after step ST44 is the obstacle detection process (application for obstacle detection) performed by the application processor 101A. In step ST44, the application processor 101A turns on the display 12 because the user U is in the viewing state. Then, the process proceeds to step ST45.

At step ST45, the application processor 101A determines whether or not the bicycle on which the user U is riding is running. For example, the application processor 101A determines whether the bicycle on which the user U is riding is running based on the sensing results of the always-on GPS (an example of the position sensor 110) and the speed sensor used for recording the running log. determine whether or not If the bicycle on which user U is riding is running, the process proceeds to step ST46.

In step ST46, after the second camera 13D is activated, the second camera 13D starts imaging in the normal mode. An image captured by the second camera 13</b>D in the normal mode is supplied to the camera signal processing section 140 . The camera signal processing unit 140 performs processing for detecting whether or not an obstacle is included in the image. For example, the camera signal processing unit 140 performs processing for detecting the presence or absence of an obstacle on the road included in the image. Obstacle monitoring using the second camera 13D is started by such processing. Although not shown, for example, when an obstacle is detected by the camera signal processing unit 140, the application processor 101A is notified to that effect. The application processor 101A that has received the notification notifies the user U of the presence of the obstacle using sound, display, vibration, a combination thereof, or the like.

On the other hand, if the determination result of step ST43 is the non-visible state (No), the process proceeds to step ST47. In step ST47, the application processor 101A turns off the display 12 because the user U is in the non-visible state. If the display 12 is in the off state, the off state is continued. Then, the process proceeds to step ST48.

In step ST48, since the user U has become invisible, the application processor 101A determines that there is no need to perform obstacle detection processing. Based on such determination, the application processor 101A performs control to stop imaging by the second camera 13D. Also, if the determination result in step ST45 is No, that is, if the vehicle is not running, there is no need to perform the obstacle detection process, so the process proceeds to step ST48 and the imaging by the second camera 13D ends. This modification can also provide the same effects as those of modification 1 described above.

In the above description, the first camera 13C captures an image in the visual recognition state detection mode, and the second camera 13D captures an image for performing the obstacle detection process. The camera may be reversed.

[Modification 3]
FIG. 15 is a diagram showing an appearance example of a portable device according to this modification. The portable device according to this modification is a watch-type portable device 100B. The mobile device 100B has a display 12C and a camera 13F provided above the display 12C. Also, the mobile device 100B has the same functions and configuration as the smart phone 100, for example. Of course, there may be structural differences and functional differences between the mobile device 100B and the smartphone 100 .

FIG. 16 is a flow chart showing the flow of processing performed by the mobile device 100B according to this modified example. The processing surrounded by the upper dotted line in FIG. 16 is the processing performed by the imaging element 104B, and the processing surrounded by the lower dotted line is the processing performed by the application processor 101A. The same applies to other figures. In step ST51, when there is no operation input to the mobile device 100B for a predetermined period of time, the mobile device 100B is controlled to be in a locked state in which no operation is accepted. The transition to the locked state is performed by the control unit 101 of the mobile device 100B, for example. Then, the process proceeds to step ST52.

The contents of steps ST52 to ST54 are the same as the processes of steps ST31 to ST33 in Modification 1, except that the front camera 13A is the camera 13F, so redundant description will be omitted. In addition, when the determination process of step ST54 is No, a process returns to step ST52.

If the determination result of step ST54 is Yes, the process proceeds to step ST55. The processing after step ST55 is, for example, an application (face authentication processing) that performs face authentication using the camera 13F, which is executed by the application processor 101A. In step ST55, the imaging mode of the camera 13F is changed from the visible state detection mode to the normal mode under the control of the application processor 101A. Note that when the visual recognition state is detected in step ST54, the system controller 131 of the imaging device 104B may autonomously switch the imaging mode from the visual recognition state detection mode to the normal mode. Also, the application processor 101A turns on the display 12C. Then, the process proceeds to step ST56.

At step ST56, a high-quality image is obtained by normal mode imaging by the camera 13F. Based on the image, for example, the camera signal processing unit 140 performs face authentication processing. A known process can be applied as the face authentication process. For example, a learning model obtained by machine learning is used to perform face authentication processing for detecting whether or not a face included in an image is a registered face. Since a high-quality image can be obtained by imaging in the normal mode of the camera 13F, face authentication can be performed accurately. If face authentication is successful in the process of step ST56 (Yes), the process proceeds to step ST57.

In step ST57, the mobile device 100B is unlocked because the authentication is successful. Then, the process proceeds to step ST58 and the process ends. If the face authentication is not established in the process of step T56 (if No), the process returns to step ST52, and the processes after step ST52 are repeated.

For example, if the face authentication process is performed when the movement of the subject is detected based on the image obtained by the camera 13F, the face authentication process will be performed even if the movement of a person other than the user U is detected. , power may be wasted. Further, when a person is detected based on an image obtained by the camera 13F and face authentication processing is performed with the detection of the person as a trigger, the face authentication processing is performed even when the user U is not looking at the mobile device 100B, and power consumption is reduced. is wasted. In this modified example, face authentication processing is performed with the user U looking at the portable device 100B, that is, in the visible state, as a trigger, so unnecessary face authentication processing can be suppressed. As a result, power consumption of the portable device 100B can be suppressed. In addition, since the application for performing face authentication is started and executed without the user U performing an operation, convenience for the user U can be improved.

It is also conceivable to detect the twist of the wrist and the orientation of the portable device 100B with a gyro sensor, and control to turn on the display based on the detection results. However, when the user U is in a sleeping posture or the like, the posture of the mobile device 100B is different from the normal posture, so there is a possibility that the display may not turn on. However, since the visual recognition state detection process according to this modification is not affected by the attitude of the portable device 100B, it is possible to prevent such inconvenience from occurring.

In this modified example, a watch-shaped portable device has been described as an example of the portable device 100B, but this modified example can also be applied to electronic devices such as smartphones.

[Modification 4]
17A to 17D are diagrams for explaining the content of this modification. As shown in FIGS. 17A to 17D, the electronic device according to this modification is a television device 100C. The television device 100C has, for example, a display 12D, and further has a camera 13G provided at a predetermined position (for example, near the upper center) of a frame surrounding the display 12D. Also, the television device 100C has an internal configuration similar to that of a known television device. The camera 13G has an imaging device 104B and is capable of executing viewing state detection processing. For example, when control is performed to turn on the main power supply of the television device 100C (for example, when the television device 100C is connected to a commercial power supply), visual recognition state detection processing using the camera 13G is started.

For example, as shown in FIGS. 17A and 17D, when the user U is viewing the television device 100C, the viewing state of the user U is detected by the same viewing state detection processing as in the first modification. be. When the viewing state is detected, for example, the television device 100C is turned on. After that, the imaging mode of the camera 13G transitions from the visual recognition state detection mode to the normal mode. Such transition control may be performed autonomously by the imaging element 104B, or may be performed under the control of the control unit of the television apparatus 100C. Then, the application processor 101A of the television device 100C performs a process of authenticating the face of the user U based on the image obtained by imaging in the normal mode. Since a high-quality image obtained by imaging in the normal mode is used, face authentication processing can be performed accurately.

Face authentication processing includes, for example, processing for matching the face of user U with the face of a registered user. If the user U is a registered user, the settings of the television device 100C are controlled to correspond to the registered user. Then, according to the settings, the content via the television broadcast or the network is reproduced. If the user U is not a registered user, default settings are used to reproduce television broadcasts and content via networks. Settings of the television apparatus 100C include volume, brightness, luminance, color tone, and the like. Note that after the setting process of the television apparatus 100C is completed, for example, the imaging mode of the camera 13G is switched from the normal mode to the visual recognition state detection mode, and imaging is performed in the visual recognition state detection mode. In the visible state detection process using the image obtained by the imaging, if the user U is in the non-visible state (for example, if the non-visible state continues for a certain period of time), the power of the television device 100C is controlled to be turned off. be done.

A method is also conceivable in which the presence or absence of a person is detected by a human sensor, and the power of the television device 100C is controlled on/off according to the detection result. However, in the method using the human sensor, as shown in FIGS. 17B and 17C, even when the user U is not watching the television device 100C, the power of the television device 100C is controlled to be turned on. Electric power may be wasted. According to this modification, when the user U is not viewing the television device 100C and is in a non-visible state, the power of the television device 100C is not controlled to be turned on, thereby preventing the above-described inconvenience from occurring.

[Modification 5]
In recent years, in public spaces such as outdoors, shops, stations, and parks (hereinafter collectively referred to as public spaces, etc.), large displays are used to transmit information. Such information transmission is called digital signage. This modification is an example in which the present disclosure is applied to digital signage.

As shown in FIG. 18, a relatively large display device 100D is installed in a public space or the like. The display device 100D has a display 12H. The display device 100D also has a camera 13H provided substantially in the center of the upper frame. The display device 100D has functions similar to those of the smart phone 100, for example.

For example, when the display device 100D is connected to the power supply, the camera 13H takes an image in the viewing state detection mode. Visual recognition state detection processing is performed in the display device 100D based on an image obtained by imaging. Content reproduction control is performed according to the detection result.

As shown in FIG. 19, when the viewing state is not detected as a result of the viewing state detection process, that is, when there is no user looking at the display 12H, the screen of the display 12H is controlled to be turned off. The standby content may be reproduced on the display 12H instead of turning off the screen of the display 12H. When the viewing state is detected as a result of the viewing state detection process, that is, when there is a user viewing the display 12H, the main content is reproduced on the display 12H. The main content is, for example, preset content. The viewing state detection process is also performed while the main content is being reproduced on the display 12H. When the viewing state is no longer detected during playback of the main content, the user U who has been viewing the main content has stopped viewing the display 12H, so control is performed to turn off the display 12H. Instead of turning off the display 12H, standby content may be played. Note that the display 12H is turned off even when the reproduction of the main content is finished. The standby content may be played instead of turning off the display 12H.

Although the above example is an example in which the operation mode of the camera 13H does not change, in this modified example, the operation mode of the camera 13H may transition from the visual recognition state detection mode to the normal mode.

For example, as shown in the flowchart of FIG. 20, the visual recognition state detection process is performed by the process from step ST61 to step ST63. The content of the viewing state detection process is the same as the process from step ST31 to step ST33 in Modification 1, except that the camera 13H is used instead of the front camera 13A. In addition, when the determination process of step ST63 is No, a process returns to step ST61.

When the determination in step ST63 is Yes, that is, when the visual recognition state is detected, the mode of the camera 13H transitions from the visual recognition state detection mode to the normal mode. Then, the process proceeds to step ST64. In step ST64, the application processor 101A performs an application for recognizing attributes of the user U viewing (browsing) the display 12H. For example, the application processor 101A performs attribute recognition processing for recognizing attributes of the user U based on an image captured by the camera 13H in normal mode. Although the method of attribute recognition processing is not limited to a specific method, for example, there is a method of applying a learning model obtained by DNN. After the attribute of user U is recognized, the process proceeds to step ST65.

In step ST65, an attribute recording process for recording user U's attributes is performed. For example, the attributes of user U are recorded in an appropriate memory within display device 100D. The attributes of the user U include gender, age, etc., which are the attributes of the user U themselves, and the time spent viewing the main content. The attributes may include attributes that are not limited to a specific user U (for example, the number of times the main content has been viewed, etc.). Note that the attributes obtained by the attribute recognition process may be transmitted to, for example, a server on the network without being recorded. In addition to the attribute recognition process, control corresponding to recognized attributes may be included in the application executed by the application processor 101A. More specifically, content reproduction control according to recognized attributes and setting processing related to content reproduction control may be included. For example, when the age of the user U is recognized as an attribute, the age-appropriate main content may be reproduced. For example, when the age of the recognized user U is small, an animation is displayed on the display 12H. Further, when the age of the recognized user U is large, control is performed to increase the volume, which is one of the setting processes related to content reproduction control. In addition, the luminance, color tone, brightness, character size, etc. of the display 12H may be adjusted according to the attribute of the user U recognized.

According to this modification, by performing the attribute recognition process, it is possible to acquire the age group and degree of interest of the user viewing the content (for example, an advertisement) displayed on the display 12H. be able to analyze the effects. Although the attribute recognition process consumes a predetermined amount of power or more, in the present modification, the attribute recognition process is performed when the visible state is detected, so power consumption in the display device 100D can be suppressed.

Instead of digital signage, this modification can also be applied to paper media, such as postings on bulletin boards. As shown in FIG. 21A, for example, notices are posted on a bulletin board 71 . An imaging device 100I is attached near the center of the upper frame of the bulletin board 71 . As shown in FIG. 21B, the imaging device 100I has, for example, a box shape and has a camera 13I on one predetermined surface. The imaging device 100I is a device that can be attached to an appropriate location on the bulletin board 71 using a thumbtack or the like, and has functions similar to those of the smartphone 100, for example.

Visual recognition state detection processing is performed based on an image captured by the camera 13I in the recognition state detection mode. When the visual recognition state is detected as a result of the visual recognition state detection process, the imaging mode of the camera 13I transitions from the visual recognition state detection mode to the normal mode. For example, the attribute recognition process described above is performed based on the image obtained by imaging in the normal mode. As a result of the attribute recognition processing, for example, the attribute of the user who viewed the notice, the number of times the notice was viewed, and the like are obtained. The information obtained can be used to analyze the effectiveness of the post and the degree of interest in the post.

Also, the postings in the above explanation may be exhibits. For example, as shown in FIG. 22, a robot 72, which is an example of an exhibit, is placed on the exhibit device 100J. The display device 100J has a columnar portion 81 and a sheet portion 82 . The sheet portion 82 can be wound around the columnar portion 81, for example. A camera 13J is provided substantially in the center of the columnar portion 81 .

Visual recognition state detection processing is performed based on an image captured by the camera 13J in the recognition state detection mode. When the visual recognition state is detected as a result of the visual recognition state detection process, the imaging mode of the camera 13J transitions from the visual recognition state detection mode to the normal mode. For example, the attribute recognition process described above is performed based on the image obtained by imaging in the normal mode. As a result of the attribute recognition processing, for example, the interest level such as the attribute of the user who viewed the exhibit and the number of times the exhibit was viewed can be obtained. Using the obtained information, it is possible to analyze information such as what kind of users are interested in what kinds of exhibits.

[Other Modifications]
The application executed by the application processor 101A corresponds to a part of the human body and a predetermined object, and includes information based on image data obtained by the imaging element 104B and sensors different from the imaging element 104B (for example, the position sensor 110 and the like). It may be determined based on at least one of the information obtained from the sensor 111). Information based on the image data obtained by the imaging device 104B includes, for example, motion vector information. The information obtained from the position sensor 110 and the sensor 111 includes, for example, at least one of position information, information about tilt of the smartphone 100, information about illuminance, and time information.

A combination of a part of the human body and a predetermined object for starting one application and another application may be similar. For example, as described in one embodiment, depending on the number of fingers, an alarm may be set, or the operation mode of the smartphone 100 may be set. In this case, by further referring to the information obtained from the sensor 111, it is possible to activate an application that meets the user's intention. For example, when the illuminance is low, when the time information is at night, when the location information indicates home, when the smartphone 100 is placed, and when a finger gesture is made, an alarm for the user to wake up is generated. likely to set. Therefore, the application to be activated is determined as an alarm. Further, when the illumination is high, when the time information is in the daytime, when the location information indicates a company, and when the smartphone 100 is being held, and a finger gesture is made, the user does not use the smartphone 100 at work. likely to be in use. Therefore, the application to be activated is determined as the mode setting control of the smart phone 100 . Such determination is made by the application processor 101A connected to the sensor 111, for example. Further, when a medicine tablet is detected, and further, when the movement vector detects that the position of the medicine has moved back and forth or up and down, in other words, when an action of taking medicine is detected, the ingestion of the medicine is detected. An application for registering the history may be activated. Further, based on the location information, the place of taking medicine may be registered.

If the application to be activated is not uniquely determined and there are multiple candidates, a screen prompting the user to make a selection may be displayed on the display 12 . Moreover, when an application is automatically started, the application may differ from the user's intention. In such a case, the launched application may be canceled. Cancellation may be performed by voice input, or may be performed by gesture for cancellation. A gesture for cancellation may be detected by imaging in the detection mode. Further, when an application is automatically activated, a notification may be made to confirm whether or not the application meets the user's intention.

A notification requesting an additional action may be made when executing the application. The request for additional action is, for example, a message such as "If you are allowed to register, please touch the confirm button." Such additional action request may be displayed on the display 12, or may be notified by sound or vibration. The additional action may be a movement in the direction of time, an action of drawing a predetermined pattern in the air, or an action for fingerprint authentication, voiceprint authentication, password authentication, etc. good. From a security point of view, the additional operation is preferably an operation known only to the user. Examples of motions in the time direction include gestures that change over time (e.g. rock, paper, scissors, paper), and objects corresponding to predetermined objects (e.g. medicine) moving vertically or in perspective. It may be an operation to move.

In the embodiment described above, part of the processing performed by the detection unit 139 and the system controller 131 may be performed outside the imaging device 104B. For example, part of the processing performed by the detection unit 139 and the system controller 131 may be performed by the control unit 101 and the application processor 101A. In this way, it is possible to appropriately change which processing is performed by which functional block.

Details of a part of the human body photographed in recognition mode may be recognized by the system controller 131 . Details of parts of the human body include facial expressions and wrinkles on the hands and face. An application to be activated may be determined in consideration of these.

In the above-described embodiment, an example in which the imaging element 104B performs imaging in any one of the moving object detection mode, detection mode, recognition mode, and normal mode has been described, but the present invention is not limited to this. For example, moving object detection may be performed based on image data obtained in the detection mode without the moving object detection mode. Also, the application to be activated may be determined based on the image data in the detection mode.

The processing according to the present disclosure can be configured as a control method or a program that causes a computer to execute the control method. The program can be distributed via a server or the like and installed in an electronic device such as a smart phone. The content of one embodiment and the content described in the modification can be appropriately combined without departing from the gist of the present disclosure.

It should be noted that the effects described in this specification are only examples and are not limited, and other effects may also occur.

Note that the present disclosure can also be configured as follows.
(1)
an imaging device that performs imaging in a first mode or in a second mode that consumes less power than the first mode;
and an application processor that executes an application corresponding to the part of the human body and the predetermined object when the part of the human body and the predetermined object are detected by imaging the imaging element in the second mode. Electronics.
(2)
The electronic device according to (1), wherein the predetermined object includes a part of a human body different from the part of the human body.
(3)
The electronic device according to (1) or (2), wherein the application processor recognizes the part of the human body as an operation instruction for the predetermined object, and executes the application according to the recognition result.
(4)
The electronic device according to any one of (1) to (3), wherein the predetermined object is imaged by the imaging device in the first mode.
(5)
The electronic device according to any one of (1) to (4), wherein the application is an application that registers information related to the predetermined object.
(6)
(5) The electronic device according to (5), wherein the information related to the predetermined object is information related to a usage history of the predetermined object.
(7)
The electronic device according to (5) or (6), wherein the predetermined object is an object held by the part of the human body.
(8)
The electronic device according to (1), wherein the predetermined object is a human body part different from the human body part, and the application processor executes an application corresponding to a combination of the human body parts.
(9)
The electronic device according to (8), wherein the human body part and the different human body part are a right hand part and a left hand part, respectively.
(10)
Execution of an application that corresponds to the part of the human body and the predetermined object and is determined based on at least one of information based on image data obtained by the imaging device and information obtained from a sensor different from the imaging device. The electronic device according to any one of (1) to (9).
(11)
(10) The electronic device according to (10), wherein the information obtained from the sensor includes at least one of information on inclination of the electronic device, information on illuminance, position information, and time information.
(12)
The electronic device according to any one of (1) to (11), wherein a notification requesting an additional operation is made when executing the application.
(13)
having a display;
The electronic device according to (12), wherein the additional operation request is displayed on the display.
(14)
The electronic device according to (12) or (13), wherein the additional motion is motion in the time direction.
(15)
The electronic device according to any one of (12) to (14), wherein the additional operation is an operation for authentication.
(16)
The electronic device according to any one of (1) to (15), wherein the predetermined object is identified by multi-wavelength sensing.
(17)
The power consumption in the second mode becomes smaller than the power consumption in the first mode by making the imaging parameter for the imaging in the second mode smaller than the imaging parameter for the imaging in the first mode. The electronic device according to any one of (1) to (16).
(18)
The electronic device according to (17), wherein the imaging parameters include at least one of resolution, color gradation, imaging area, and frame rate.
(19)
The imaging device performs imaging in the first mode or in a second mode that consumes less power than the first mode,
an application processor executing an application corresponding to the part of the human body and the predetermined object when the part of the human body and the predetermined object are detected by imaging the imaging element in the second mode; Method.
(20)
The imaging device performs imaging in the first mode or in a second mode that consumes less power than the first mode,
an application processor executing an application corresponding to the part of the human body and the predetermined object when the part of the human body and the predetermined object are detected by imaging the imaging element in the second mode; A program that makes a computer perform a method.
(21)
an imaging device that performs imaging in a first mode or in a second mode that consumes less power than the first mode, and detects whether or not the imaging device is in a visible state according to imaging results in the second mode; ,
and an application processor that executes a predetermined application when the result of the detection is the visible state.
(22)
(21) The electronic device according to (21), wherein when the result of the detection is the visible state, the operation mode of the imaging element is changed to the first mode.
(23)
(22) The electronic device according to (22), wherein the predetermined application includes a process of recognizing an attribute of the visually recognizing person in the visually recognizing state based on the imaging result in the first mode.
(24)
(23) The electronic device according to (23), wherein the predetermined application includes a content reproduction control process corresponding to the attribute and a setting process related to the content reproduction control.
(25)
The electronic device according to (21) or (22), wherein, when the result of the detection is the visible state, the application processor refers to the sensing result of a sensor different from the imaging element and executes the predetermined application. .
(26)
Having another imaging device different from the imaging device,
(25) The electronic device according to (25), wherein the predetermined application includes a process of detecting an obstacle based on the image obtained from the other imaging device.
(27)
The imaging device performs imaging in the first mode or in a second mode that consumes less power than the first mode, and detects whether or not it is in a visible state according to the imaging result in the second mode. death,
A control method in which an application processor executes a predetermined application when a result of the detection is the visible state.
(28)
The imaging device performs imaging in the first mode or in a second mode that consumes less power than the first mode, and detects whether or not it is in a visible state according to the imaging result in the second mode. death,
A program that causes a computer to execute a control method in which an application processor executes a predetermined application when the result of the detection is the visible state.

12... Display 100... Smart phone 101... Control unit 101A... Application processor 104B... Image sensor 139... Detection unit

Claims

an imaging device that performs imaging in a first mode or in a second mode that consumes less power than the first mode;
and an application processor that executes an application corresponding to the part of the human body and the predetermined object when the part of the human body and the predetermined object are detected by imaging the imaging element in the second mode. Electronics.
The electronic device according to claim 1, wherein the predetermined object includes a part of a human body different from the part of the human body.
The electronic device according to claim 1, wherein the application processor recognizes the part of the human body as an operation instruction for the predetermined object, and executes the application according to the recognition result.
2. The electronic device according to claim 1, wherein an image of the predetermined object is captured by the imaging device in the first mode.
The electronic device according to claim 1, wherein the application is an application that registers information related to the predetermined object.
The electronic device according to claim 5, wherein the information related to the predetermined object is information related to the usage history of the predetermined object.
The electronic device according to claim 5, wherein the predetermined object is an object held by the part of the human body.
The electronic device according to claim 1, wherein the predetermined object is a human body part different from the human body part, and the application processor executes an application corresponding to a combination of the human body parts.
The electronic device according to claim 8, wherein the human body part and the different human body part are a right hand part and a left hand part, respectively.
Execution of an application that corresponds to the part of the human body and the predetermined object and is determined based on at least one of information based on image data obtained by the imaging device and information obtained from a sensor different from the imaging device. The electronic device according to claim 1.
The electronic device according to claim 10, wherein the information obtained from the sensor includes at least one of information regarding inclination of the electronic device, information regarding illuminance, position information, and time information.
The electronic device according to claim 1, wherein a notification requesting an additional operation is provided when executing the application.
having a display;
13. The electronic device of claim 12, wherein the additional action request is displayed on the display.
The electronic device according to claim 12, wherein the additional motion is motion in the time direction.
The electronic device according to claim 12, wherein the additional operation is an operation for authentication.
The electronic device according to claim 1, wherein identification of the predetermined object is performed by multi-wavelength sensing.
The power consumption in the second mode becomes smaller than the power consumption in the first mode by making the imaging parameter for the imaging in the second mode smaller than the imaging parameter for the imaging in the first mode. The electronic device according to claim 1.
The electronic device according to claim 17, wherein the imaging parameters include at least one of resolution, color gradation, imaging area, and frame rate.
The imaging device performs imaging in the first mode or in a second mode that consumes less power than the first mode,
an application processor executing an application corresponding to the part of the human body and the predetermined object when the part of the human body and the predetermined object are detected by imaging the imaging element in the second mode; Method.
The imaging device performs imaging in the first mode or in a second mode that consumes less power than the first mode,
an application processor executing an application corresponding to the part of the human body and the predetermined object when the part of the human body and the predetermined object are detected by imaging the imaging element in the second mode; A program that makes a computer perform a method.
an imaging device that performs imaging in a first mode or in a second mode that consumes less power than the first mode, and detects whether or not the imaging device is in a visible state according to imaging results in the second mode; ,
and an application processor that executes a predetermined application when the result of the detection is the visible state.
22. The electronic device according to claim 21, wherein control is performed to change the operation mode of the imaging element to the first mode when the result of the detection is the visible state.
23. The electronic device according to claim 22, wherein the predetermined application includes a process of recognizing an attribute of the visually recognizable person in the visually recognizable state based on the imaging result in the first mode.
24. The electronic device according to claim 23, wherein the predetermined application includes content reproduction control processing corresponding to the attribute and setting processing related to content reproduction control.
22. The electronic device according to claim 21, wherein, when the result of the detection is the visible state, the application processor refers to the sensing result of a sensor different from the imaging element and executes the predetermined application.
Having another imaging device different from the imaging device,
26. The electronic device according to claim 25, wherein the predetermined application includes processing for detecting an obstacle based on the image obtained from the other imaging device.
The imaging device performs imaging in the first mode or in a second mode that consumes less power than the first mode, and detects whether or not it is in a visible state according to the imaging result in the second mode. death,
A control method in which an application processor executes a predetermined application when a result of the detection is the visible state.
The imaging device performs imaging in the first mode or in a second mode that consumes less power than the first mode, and detects whether or not it is in a visible state according to the imaging result in the second mode. death,
A program that causes a computer to execute a control method in which an application processor executes a predetermined application when the result of the detection is the visible state.