JP2013246471A - Operation reaction force providing system for portable equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide operation reaction force on a control equipment side when displaying a display object of a selection object necessary for portable equipment on a display screen of the control equipment to perform selection operation, and to reduce a memory amount and a control operation load on the control equipment side.SOLUTION: In an on-vehicle device 30 capable of communicating with a portable terminal 20, when a user operates an operation lever 10, a display control unit 315 moves a cursor on a display screen of a display unit 304. In a situation where the cursor moves in a reaction force providing region out of a display region of a display object, a reaction force control unit 312 activates an operation reaction force providing mechanism 311. Then, operation reaction force to be resistance against the operation of the operation lever 10 occurs.

Description

  The present invention relates to an operation reaction force application system for a portable device.

  In a mobile device such as a smart phone, a user performs an input operation by sliding or touching a finger on the display panel of the mobile device. Cannot be recognized by touch. Therefore, when there are a plurality of display objects to be selected on the display screen, it is not obvious from the tactile sensation of which display object to be selected by the user, and it is necessary to operate the display screen while constantly checking with the eyes.

On the other hand, in recent years, communicable mobile devices are communicated with other control devices such as on-vehicle devices, and on the on-vehicle device side, when executing any application software of the mobile device, the mobile device and the on-vehicle device operate in cooperation, It is conceived that the input / output of the portable device is assumed by the device.
Therefore, when a mobile device and a control device are operated in cooperation to execute any application software on the mobile device, the control device selects a display object to be selected on the display screen according to the operation of the mobile device. In some cases, it is desired that an operation reaction force can be applied.
The device control device disclosed in Patent Document 1 has a haptic commander that can communicate with various devices such as portable devices and can recognize each device, and displays information on the devices recognized by the haptic commander. And a control device that controls the device and the monitor, and is configured to obtain a different operational feeling for each device to be controlled.

  However, with the technique of Patent Document 1, only a different operational feeling is provided for each device to be controlled, and an operational feeling capable of distinguishing each of the plurality of display objects on the display screen cannot be obtained. Further, in Patent Document 1, since control software for an assumed device is stored in the haptic commander, the control software to be stored increases as the number of communicable devices increases, and a lot of memory is required.

JP 2004-120577 A

  The present invention has been made in view of the above circumstances, and its purpose is to display a reaction object to be applied when a mobile device and a control device are operated in cooperation to execute arbitrary application software of the mobile device. Operational force for mobile devices that can be applied on the control device's display screen and can be applied on the control device side, and that can reduce the amount of memory and control load on the control device. To provide a system.

In the first aspect of the invention, the portable device drawing control means is based on the display data of the display object including the display object to be applied with the reaction force and the display object not to be applied with the reaction force displayed on the display device for the mobile device. Displays various display objects. Then, in the portable device, the extraction means extracts the reaction force application region based on the reaction force application target display object when the display screen of the portable device display means is switched. And a data transmission control means transmits the said display data and reaction force provision area | region data to a control apparatus.
Then, in the control device, the display control unit supplies the display data to the control device drawing control unit, causes the display unit for the control device to display the display object, and displays the display object on the control device display unit. An indicator part for instruction is displayed. This display control means moves the indicator section on the display screen of the control device display means in accordance with the operation of the operator.

  When the user operates the operation element, the display control unit moves the index unit on the display screen of the control device based on the operation of the operation element. When the force application region is moved, the reaction force control unit operates the operation reaction force application unit. Then, an operation reaction force that becomes resistance to the operation of the operation element is generated, and the user feels the operation reaction force as a tactile sensation. , It can be recognized that the indicator section indicates the display object.

In addition, since the mobile device is provided with the storage device for mobile device that stores the reaction force application area data in the display object to which the reaction force is applied, there is no need to store the reaction force application region data in the control device. The storage capacity of the storage means can be reduced, and the storage control process can be reduced.
Furthermore, since the display data and the reaction force application area data are transmitted to the control device by the data transmission control unit in the portable device, the control device can receive and display the display data transmitted from the data transmission control unit. Since it is good, it is not necessary to hold (store) display data on the control device side, and this can also reduce the storage capacity of the storage means in the control device and reduce the storage control processing.

The functional block diagram of the operation reaction force provision system for portable devices by embodiment of this invention A perspective view of the inside of the vehicle showing the arrangement of the control lever The figure which shows schematic structure of the operation reaction force provision mechanism part It is a flowchart which shows operation | movement of the operation reaction force provision system for portable devices, (a) is a flowchart by the side of a portable terminal, (b) is a flowchart by the side of vehicle equipment. Figure of display screen showing display example Diagram showing display data It is a figure which shows notionally the reaction reaction force pattern of the display object H1, (a) is a top view, (b) is a BB arrow line view of (a), (c) is CC of (a). Arrow view It is a figure which shows notionally the reaction force pattern of the display object H2, (a) is a top view, (b) is DD arrow directional view of (a). Diagram showing reaction force application area data (A) is a figure for comparing the screen size of the display part of a portable terminal, and the display part of a vehicle-mounted apparatus, (b) is a figure of the display screen in the display part of a vehicle-mounted apparatus. (A) is a figure for demonstrating the screen size of the display part of a portable terminal, (b) is a figure of the display screen in case the screen sizes of the display part of a vehicle-mounted apparatus differ. FIG. 7 equivalent view conceptually showing another example of the operation reaction force pattern of the display object H1. FIG. 8 equivalent view conceptually showing another example of the operation reaction force pattern of the display object H2.

  Embodiments of the present invention will be described below with reference to FIGS. As shown in FIG. 1, an operation reaction force imparting system 10 for a mobile device is constructed from a mobile terminal 20 such as a smart phone corresponding to a mobile device and an in-vehicle device 30 including a navigation device corresponding to a control device. Has been. The portable terminal 20 includes a control unit 201 corresponding to a portable device control unit, a wireless communication unit 202, a wired communication unit 203, an operation input unit 204, a display unit 205 corresponding to a portable device display unit, and a portable device drawing control unit. A drawing control unit 206 corresponding to the storage unit 207, a storage unit 207 corresponding to the portable device storage unit, a sound processing unit 208, a program providing unit 209 corresponding to the program providing unit, an extraction unit 210 corresponding to the extracting unit, and a data transmission control unit A data transmission control unit 211 corresponding to the reduction rate calculation unit 212 corresponding to the reduction rate calculation unit and a data change unit 213 corresponding to the data change unit are configured.

  On the other hand, the in-vehicle device 30 corresponds to a control unit 301 corresponding to a control device control unit, a wireless communication unit 302, a wired communication unit 303, a display unit 304 corresponding to a control device display unit, and a control device drawing control unit. Drawing control unit 305, audio processing unit 306, storage unit 307 corresponding to control device storage unit, audio output unit 308, vehicle I / F (input / output interface) 309, operation lever 310, operation reaction force application unit Operation reaction force application mechanism 311, reaction force control unit 312 corresponding to reaction force control means, operation direction detection mechanism 313, operation determination unit 314, display control unit 315 equivalent to display control means, and operation equivalent to operation information transmission means An information transmission unit 316 is provided. The control unit 301, the wireless communication unit 302, the wired communication unit 303, the display unit 304, the drawing control unit 305, the voice processing unit 306, the storage unit 307, and the voice output unit 308 are some of the components of the navigation device 317. It is composed. In addition, the navigation device 317 includes a position detector, a map data storage unit, and the like.

  The control unit 201 of the portable terminal 20 includes a well-known microcomputer having a CPU, a RAM, a ROM, an I / O bus, and the like (not shown). The control unit 201 controls the operation of the entire mobile terminal 20 according to a computer program stored in the ROM or the storage unit 207. In addition, the control unit 201 executes a cooperation program with reaction force for mobile devices prepared in advance separately from the computer program, so that the program providing unit 209, the extraction unit 210, the data transmission control unit 211, It also functions as the reduction ratio calculation unit 212 and the data change unit 213.

The wireless communication unit 202 includes communication means such as Bluetooth (registered trademark) and Wi-Fi (registered trademark), and can communicate with a wireless communication unit 302 of the in-vehicle device 30 described later.
The wired communication unit 203 can communicate with a wired communication unit 303 of the in-vehicle device 30 described later via a USB cable or a dedicated cable. The operation input unit 204 includes various switches such as a touch panel switch provided on a display screen of a display unit 205 described later and a mechanical switch provided around the display unit 205. The operation input unit 204 outputs an operation detection signal to the control unit 201 according to various switch operations by the user. The control unit 201 analyzes the operation detection signal input from the operation input unit 204 to specify the operation content of the user, and executes various processes based on the specified operation content.

  The display unit 205 includes, for example, a liquid crystal display or an organic EL display. When the drawing control unit 206 is controlled based on a display command signal from the control unit 201, the drawing control unit 206 performs various operations. Display information of. The display surface of the display unit 205 is provided with a touch panel switch configured by a known pressure-sensitive method, electromagnetic induction method, capacitance method, or a combination thereof. The display unit 205 displays an operation input screen for inputting an operation on the content and an output screen for outputting the execution content of the content.

  The storage unit 207 is configured by a non-volatile storage medium such as a memory card, for example, and realizes a cooperation function for executing content in cooperation with various computer programs and content programs and external devices and terminals. Various programs such as cooperative applications and data used in each program are stored. As the data, as display objects to be displayed on the display unit 205, display data of a display object that is a reaction force application target and a display object that is not a reaction force application target is stored, and a reaction in the display object that is the reaction force application target is stored. Force-giving area data is stored.

The drawing control unit 206 is composed of, for example, a one-chip microcomputer, and the display data (display coordinate data regarding the display position and display shape) of the display object displayed on the display screen of the display unit 205 given from the control unit 201. And the display unit 205 is controlled based on data such as display color.
The voice processing unit 208 is connected to a microphone and a speaker (not shown) and has a known voice input function and voice (including music) output function.
On the other hand, the control unit 301 of the in-vehicle device 30 is configured by a known microcomputer having a CPU, a RAM, a ROM, an I / O bus, and the like (not shown). The control unit 301 controls the overall operation of the in-vehicle device 30 according to a computer program stored in the ROM or the storage unit 307, and also executes a cooperation program with reaction force for control equipment provided from the mobile terminal 20. . By executing the control device reaction force cooperation program, the display control unit 315 and the operation information transmission unit 316 function.

The wireless communication unit 302 includes communication means such as Bluetooth (registered trademark) and Wi-Fi (registered trademark), and can communicate with the wireless communication unit 202 of the mobile terminal 20. The wired communication unit 303 can communicate with the wired communication unit 203 of the mobile terminal 20 via a USB cable or a dedicated cable. Note that the communication means may be either wired or wireless.
The display unit 304 includes, for example, a liquid crystal display or an organic EL display. When the drawing control unit 305 is controlled based on a display command signal from the control unit 301, the display control unit 305 performs various operations. Display information of. The display surface of the display unit 304 is provided with a touch panel switch configured by a known pressure-sensitive method, electromagnetic induction method, capacitance method, or a combination of these.

  The drawing control unit 305 is controlled by the display control unit 315, and is based on display data (display coordinate data related to display position and display shape) of display objects displayed on the display unit 304. Control drawing. Further, the drawing control unit 305 moves and displays a cursor C (see FIG. 5) as an index unit on the display screen of the display unit 304 in accordance with an operation of an operation lever 310 described later.

The voice processing unit 306 outputs voice and music from the voice output unit 308. The audio output unit 308 is composed of a speaker. Note that a microphone (not shown) is connected to the voice processing unit 306 and has a voice input function.
The storage unit 307 is configured by a non-volatile storage medium such as an EEPROM, for example. The storage unit 307 is a linkage that realizes a linkage function for executing content in cooperation with various computer programs and content programs, and external devices and terminals. Various programs such as applications and data used in each program are stored. Or temporarily store.

  As shown in FIG. 2, the operation lever 310 is provided on the center console so as to be tiltable from the neutral position to the entire circumference direction. The operation lever 310 is provided with an operation reaction force applying mechanism 311 as an operation reaction force applying means. Is provided. As shown in FIG. 3, the operation reaction force application mechanism 311 includes an X axis motor 311x that applies an operation resistance force (operation reaction force) in the X axis direction (right direction, left direction), and a Y axis direction (forward direction). , A Y-axis motor 311y that applies an operation resistance force (operation reaction force) in the rearward direction, and a motor control unit 311s that drives and controls these motors 311x and 311y. The motor control unit 311 s of the operation reaction force applying mechanism 311 is controlled by the reaction force control unit 312.

  Further, the operation lever 310 includes an operation direction detection mechanism 313 including an X-axis direction displacement sensor 313x and a Y-axis direction displacement sensor 313y (for example, both are rotary encoders) that detect an operation direction and an operation displacement of the operation lever 310. ing. The operation direction and operation displacement data detected by the operation direction detection mechanism 313 is given to the control unit 301 via the vehicle I / F 309, and the control unit 301 detects the cursor C based on the operation direction and operation displacement data. The display coordinates are calculated and given to the drawing control unit 305. The drawing control unit 305 moves and displays the cursor C on the display unit 304 based on the display coordinates of the cursor C.

Further, an operation determination unit 314 including, for example, a tact switch is provided below the operation lever 310. When the operation lever 310 is pressed, an operation input for the display object indicated by the cursor at the present time is provided. A switch-on input is input to the control unit 301.
The reaction force control unit 312 can communicate with the control unit 301 via the vehicle I / F 309. The reaction force control unit 312 controls the operation reaction force application mechanism 311 when the cursor C corresponding to the operation direction and operation displacement of the operation lever 310 enters the reaction force application region.

  Here, in FIG. 5, an example of a display object displayed on the display screen 205a of the display unit 205 will be described. The display objects H1 and H2 shown on the display screen 205a both show graphic data, and the content of the drawing data which is the display data of the display object H1 is, for example, a graphic as shown in FIG. ID “00001”, shape type “rectangular”, function type “switch box”, size “width 10 pixels × length 5 pixels”, drawing position coordinate K1 (X1, Y1), and drawing color mode “monochrome” The drawing color is (R1, G1, B1).

  Further, as shown in FIG. 6B, the content of the drawing data that is the display data of the display object H2 is, for example, a graphic ID “00002”, a shape type “circle”, a function type “scroll area”, The size is “radius 10 pixels”, the drawing position coordinate center K2 is (X2, Y2), the drawing color mode is “gradation”, the drawing color is “center (R2, G2, B2), the outer periphery (R3, G3, B3)” The “scroll area” is the display object H2 of the “scroll area” for the operation range of the operation lever 310 when scrolling in a map display or scrolling a large number of selection items (music title items, etc.). Used to regulate.

Each display object is set in advance as to whether or not it is a display object to which an operation reaction force is applied according to the function type, and the types “switch box” and “scroll area” described above are set. It is preset that the display object is a display object to which an operation reaction force is applied.
Further, reaction force application areas in the display objects of the display screen are set in advance according to the function types described above. That is, an image of the operation reaction force pattern of the display object H1 is shown in FIGS. 7A to 7C, and an image of the operation reaction force pattern of the display object H2 is shown in FIGS. 8A and 8B. . FIG. 9 shows reaction force application area data including operation reaction force pattern data of the display objects H1 and H2.

In FIG. 9, the reaction force application area data of the display object H1 is
Reaction force boundary outer periphery coordinate data: reference coordinates K1 (X1, Y1), size XP × YP (in this case, 10 pixels × 5 pixels)
Reaction force boundary edge coordinate data: Wa, Wb
Maximum reaction force level: T1
It has become. The reaction force boundary outer periphery coordinate data also serves as display data of the display object H1 displayed on the display unit 304.

In this case, the operation reaction force pattern is a pattern in which the strength of the reaction force is virtually flat (rectangular) within the reaction force application region, as can be seen from FIGS. 7B and 7C.
The reaction force application area data of the display object H2 is as shown in FIG.
Reaction force boundary outer periphery coordinate data: reference center coordinate K2 (X2, Y2), size = radius RP (in this case, 10 pixels)
Reaction force boundary rim coordinate data: Wc
Reaction force maximum coordinate: Wd
Maximum reaction force level: T2
It has become. The reaction force boundary outer peripheral coordinate data also serves as display data of the display object H2 displayed on the display unit 304.

  In this case, as shown in FIG. 8B, the operation reaction force pattern is a pattern in which the strength of the reaction force is virtually mountain-shaped within the reaction force application region. In particular, the operation reaction force pattern in the approach operation direction to the display object H2 is gradually increased and suddenly reduced, and the operation reaction force pattern in the separation direction with respect to the display object H2 is rapidly increased and gradually reduced. It is a pattern to do.

  The operation reaction force pattern is preset as a default, but can be set by the user on the mobile terminal 20 side. In this case, an operation reaction force pattern such as a flat shape, a mountain shape, or an arc shape is displayed on the display unit 205 so that it can be selected from the operation reaction force patterns by a screen touch operation or the like. The level and maximum level position can also be set. FIG. 12 shows another setting example of the operation reaction force pattern of the display object H1, and FIG. 13 shows another setting example of the operation reaction force pattern of the display object H2.

  Now, the operation of the operation reaction force imparting system 10 for a mobile device, the operation of the mobile terminal 20, and the operation of the in-vehicle device 30 will be described with reference to FIGS. 4 (a) and 4 (b). When a power switch (not shown) of the portable terminal 20 is turned on, the operation of FIG. 4A is started, and when the ACC switch of the vehicle is turned on, the operation of FIG. 4B is started. First, in the portable terminal 20, the control unit 201 controls the drawing control unit 206 to display a startup screen on the display unit 205 (step S10).

  In step S20, it is determined whether there is a switch input from the operation input unit 204 and the application software of the portable terminal 20 is activated. When the application software of the self-portable terminal 20 is activated, in the next step S30, the self-portable terminal 20 is brought into the vehicle, and the wireless communication unit 202 or the wired communication unit 203 is changed to the wireless communication unit 302 or the in-vehicle device 30 respectively. It is determined whether or not communication with the wired communication unit 303 is established. If it is determined that the connection with the in-vehicle device 30 is established, the process proceeds to step S40 to execute a connection confirmation process with the in-vehicle device 30.

  In the next step S50, it is determined whether or not the operation input unit 204 has performed an input operation in the cooperation mode with reaction force. If it is determined that the input operation has been performed, in step S53, the operation of the cooperative application software with reaction force for mobile terminal corresponding to the cooperation program with reaction force for mobile device is started. For example, the mobile terminal reactive application software with reaction force may be installed in advance from a provider (server). The mobile terminal cooperation application software with reaction force includes in-vehicle device reaction force cooperation application software corresponding to a control device reaction force cooperation program executed on the in-vehicle device 30 side as a separate file.

  In step S54, a display object is displayed according to the cooperative application software with reaction force for mobile terminal, and in step S55, data of the cooperative application software with reaction force for in-vehicle device is transmitted to the in-vehicle device 30 (program providing unit). 209).

  On the other hand, in the in-vehicle device 30, the operation of the normal application software of the in-vehicle device 30 is executed in the first step T10. In step T20, it is determined whether or not the mobile terminal 20 is communicably connected. If it is determined that the mobile terminal 20 is connected, connection setting and response processing with the mobile terminal 20 is executed in step T30. After this, in step T35, it waits for reception of information from the portable terminal 20 (data of cooperative application software with reaction force for in-vehicle devices), and if the information is received, the process proceeds to step T40. In step T40, it is determined from the received information whether or not the application software is in the cooperation mode with reaction force.

  If it is determined in this step T40 that the cooperation mode is with reaction force, the data of the cooperation application software with reaction force for in-vehicle devices is stored in the storage unit 307 in step T50, and in step T60, The operation of the cooperative application software with reaction force for in-vehicle devices is started. In step T <b> 70, information on the screen size (the number of horizontal pixels × the number of vertical pixels) of the display screen 304 a of the display unit 304 in the in-vehicle device 30 is transmitted to the mobile terminal 20.

  Reception of this screen size information is confirmed in the portable terminal 20 in step S80. Thereafter, the process proceeds to step S90, where the screen size of the display screen 304a of the display unit 304 of the in-vehicle device 30 is compared with the screen size of the display unit 205 of the portable terminal 20, and the reduction rate of the screen size is calculated as necessary. (Reduction rate calculation unit 212). For example, as shown in FIG. 10A, the screen size (drawing area) of the display screen 205 a of the display unit 205 of the mobile terminal 20 is, for example, horizontal 800 pixels × vertical 600 pixels, and the display unit 304 of the in-vehicle device 30. When the screen size of the display screen 304a is, for example, horizontal 1024 pixels × vertical 768 pixels, the display screen of the display unit 304 of the in-vehicle device 30 with respect to the screen size of the display screen 205a of the display unit 205 of the mobile terminal 20 Since the screen size of 304a is larger, the screen size of the portable terminal 20 is not reduced.

In this case, as shown in FIG. 10B, the display screen 205a ′ of the mobile terminal 20 on the display screen 304a of the in-vehicle device 30 matches the coordinate origin 205P with the coordinate origin 304P of the display screen 304a of the in-vehicle device 30. To display.
Also, as shown in FIG. 11A, the screen size of the display screen 205a of the display unit 205 of the mobile terminal 20 is, for example, horizontal 800 pixels × vertical 600 pixels, and as shown in FIG. When the screen size of the display screen 304a of the display unit 304 of the device 30 is, for example, horizontal 640 pixels × vertical 480 pixels (when the screen size of the display screen 304a is smaller), the reduction rate is calculated (reduction rate calculation). Part 212). In this case, the reduction ratio is 80% as a linear ratio.

  After step S90 described above, the process proceeds to step S100. In this step S100, it is determined whether or not there is a graphic data display object in the display data (in this case, graphic data and other data are included) transmitted from the control unit 201 to the drawing control unit 206. If there is a graphic data display object, in step S110, the graphic data display object is selected, and in step S120, display of the function type for which reaction force is added in advance in one or more graphic data. If there is an object (a display object to which reaction force is applied), it is extracted.

  In step S130, a reaction force application region is extracted based on the reaction force application target display object extracted in step S120 (extraction unit 210). For example, the reaction force application region shown in FIGS. 7 to 9 described above is extracted.

  Then, in the next step S140, display data relating to the display position and display size of the reaction object to which the reaction force is applied and data of the reaction force application region are changed in accordance with the reduction ratio calculated in step S90 ( Data changing unit 213). Then, the process proceeds to the next step S150. If it is not necessary to calculate the reduction ratio in step S90, the process proceeds to the next step S150 without performing this correction. In step S150, the reaction force application area data of each display object and the display data of each display object are transmitted to the in-vehicle device 30 (data transmission control unit 211).

  On the other hand, in the in-vehicle device 30, when it is determined that the display data and the reaction force application area data transmitted in step S150 are received in step T80, the process proceeds to step T90, and the display data is sent to the drawing control unit 305. Then, the drawing control unit 305 displays the cursor C on the display screen 304a as shown in FIG. 5, and also displays the display objects H1 and H2 (display control unit 315). In this case, the cursor C indicates a part (for example, the center of the screen) other than both display objects H1 and H2.

  Here, when the user operates the operation lever 310 in an appropriate direction, this is determined as “YES” in Step T100. In step T110, the cursor C moves according to the operation direction of the operation lever 310. When the operation lever 310 is further operated and the cursor C enters the inside from the outer periphery of the display object H1, for example, it is determined in step T120 that the cursor C has entered the reaction force application region, and in step T130, The operation reaction force application mechanism 311 is drive-controlled according to the reaction force application pattern (reaction force control unit 312). As a result, a reaction force acting as an operation resistance is applied to the operation lever 310. In this case, the strength of the reaction force is virtually changed to flat (uniform reaction force) within the reaction force application region. Thereby, the user first feels the operation reaction force as a tactile sensation, and can recognize that the cursor C, that is, the cursor C by the operation of the operation lever 310 has entered the outer peripheral edge of the display object H1 by the tactile sensation at hand.

When the operation lever 310 is further operated toward the center of the display object H1 against this operation reaction force, the reaction force intensity is set to a flat pattern in the reaction force application region. Are alleviated at once and reach the center of the display object H1. Thereby, the user can recognize that his / her operation indicates the inside of the display object.
In this state, for example, when the user performs a specific operation on the display object H1, for example, when the user presses the operation lever 310, the operation determination unit 314 is turned on, and in this case, the switch-on input is controlled as the operation input for the display object H1. Input to the unit 301. This switch input is determined as “YES” in step T140 for determining the operation information or the command generation, and the operation information of this switch input is transmitted to the portable terminal 20 in step T150 (operation information transmitting unit 316).

Further, when the user operates the operation lever 310 in a direction to release from the display object H1, the operation reaction force suddenly increases and then becomes a flat reaction force. It can be recognized that the user has left the object H1.
The operation information or command transmitted by the in-vehicle device 30 is determined to be received in step S160 in the portable terminal 20. In the next step S170, the received operation information or command is analyzed, and processing corresponding to the analysis result is executed.

  In the next step S180, it is determined whether or not the content of the process is a screen change. If the screen is changed, the process proceeds to step S190 and the display content of the display screen 205a of the display unit 205 of the portable terminal 20 is changed. To do. In the next step S200, it is determined whether or not the cooperation mode with reaction force is ended (whether the setting of the cooperation mode with reaction force has been canceled by the operation input unit 204 or the communication state has been canceled). If the cooperation mode with reaction force is not completed, the process returns to step S100. If the cooperation mode with reaction force ends, the cooperation mode end processing with reaction force (application software end) is executed in step S210, and the process returns to step S20.

  After step T150 in the in-vehicle device 30, whether or not the cooperation mode with reaction force has ended (whether or not the setting of the cooperation mode with reaction force has been canceled by an appropriate operation or the communication state has been released). If the cooperation mode with reaction force ends, the cooperation mode end processing with reaction force (application software end) is executed in step T170, and the process returns to step T10.

If “NO” in the step S30, the process proceeds to a step S220 to execute normal application software. If “NO” in the step S50, the normal cooperation in the cooperation mode without reaction force is performed in the step S230. Run application software. If “NO” in the step S100, the process proceeds to a step S160. If “NO” in the step T40, the normal cooperative application software in the normal cooperation mode without the reaction force is executed in a step T180.

  On the other hand, when the cursor C enters the display object H2, the reaction force control unit 312 operates to apply the operation reaction force to the operation lever 310. In this case, since the operation reaction force pattern forms a mountain shape, the operation reaction force suddenly increases, and if this is overcome, it suddenly decreases. As described above, since the operation reaction force patterns are different between the display object H1 and the display object H2, it is possible to know which display object the user is pointing to without depending on visual recognition due to the difference in the operation reaction force.

Then, when the cursor C enters the display object H2, for example, if the user performs a specific operation on the display object H2, for example, presses the operation lever 310, the operation determination unit 15 switches on as an operation input to the display object H2. An input is input to the control unit 301. As in the case of the display object H1, the switch input is determined as “YES” in step T140 for determining operation information or command generation, and the switch input operation information is transmitted to the portable terminal 20 in step T150. The
In this case, when receiving the switch-on input operation information in step S170, the portable terminal 20 analyzes the operation information content in step S170 and executes a process corresponding to this, as described above. Migrate to

  In the embodiment described above, when communication between the mobile terminal 20 and the in-vehicle device 30 is established, and the mobile terminal 20 starts the cooperative application software with reaction force for mobile device corresponding to the cooperation program with reaction force for mobile device, the mobile device 20 In the terminal 20, the program providing unit 209 transmits to the in-vehicle device 30 cooperation application software with reaction force for in-vehicle device corresponding to the cooperation program with reaction force for control device. Further, the drawing control unit 206 displays a display object on the display unit 205. And in the portable terminal 20, the extraction part 210 extracts reaction force provision area | region based on the display object of reaction force provision object, and the display data and reaction force provision area | region of the display object which the data transmission control part 211 extracted Data is transmitted to the in-vehicle device 30.

  Then, in the in-vehicle device 30, the display control unit 315 supplies the transmitted display data of the display object to the drawing control unit 305 of the in-vehicle device 30 to display the display object on the display unit 304, and the display A cursor C for indicating a display object is displayed on the unit 304. The display control unit 315 moves the cursor C on the display screen of the display unit 304 of the in-vehicle device 30 in accordance with the operation of the operator.

  When the user operates the operation lever 310, the display control unit 315 moves the cursor C on the display screen 304a of the display unit 304 of the in-vehicle device 30 based on the operation of the operation lever 310. When the state moves to the reaction force application area in the display area of the display object, the reaction force control unit 312 operates the operation reaction force application mechanism 311. Then, an operation reaction force that becomes resistance to the operation of the operation lever 310 is generated, and the user feels the operation reaction force as a tactile sensation. From the tactile sensation, it can be recognized that the cursor C indicates the display object.

Moreover, since the storage unit 207 that stores the reaction force application area data in the display object that is the reaction force application target is provided in the mobile device 20, it is not necessary to store the reaction force application area data in the in-vehicle device 30. The storage capacity of the storage unit 307 can be reduced, and the storage control process can be reduced.
Furthermore, since the display data and the reaction force application area data are transmitted to the in-vehicle device 30 by the data transmission control unit 211 in the portable device 20, the in-vehicle device 30 receives the display data transmitted from the data transmission control unit 211. Therefore, it is not necessary to store (store) the display data on the in-vehicle device 30 side, and this can also reduce the storage capacity of the storage unit 307 in the in-vehicle device 30 and also perform storage control processing. Can be reduced.

  Moreover, according to this embodiment, since the control apparatus which bears reaction force provision with respect to the portable terminal 20 is the vehicle-mounted apparatus 30, it is very convenient. That is, the mobile terminal 20 has many opportunities to bring it into the vehicle, and there are many opportunities to execute the content (for example, music playback content) of the mobile terminal 20 on the in-vehicle device 30. In this case, it is possible to operate the display object to which the operation reaction force is to be applied in the mobile terminal 20 while feeling the operation reaction force with the operation lever 310 of the in-vehicle device 30. In particular, in the in-vehicle device 30 mounted on the vehicle, the display object can be recognized at hand without much viewing the display unit 304 of the in-vehicle device 30, which is advantageous in terms of safe driving.

  In this embodiment, the function type of the display object H1 is “switch box”, and the function type of the display object H2 is “scroll area”, and both function types are different. Further, both display objects H1 and H2 have different graphic types. In this embodiment, the operation reaction force pattern in the reaction force application region is made different depending on the function type and graphic type of the display objects H1 and H2. The function type and graphic type of the display object can be recognized without looking at the screen 304a.

Note that the operation reaction force pattern in the reaction force application region may be different in at least one of the function type and the graphic type of the display objects H1 and H2. Further, when the display colors of the display objects H1 and H2 displayed on the in-vehicle device 30 are made different, the operation reaction force pattern may be made different according to the display color.
In the present embodiment, the operation reaction force pattern in the approach operation direction to the display object H2 is different from the operation reaction force pattern in the disengagement direction with respect to the display object H2, so that the user can display the display object H2. On the other hand, whether the cursor C has entered or left the display object H2 can be easily recognized without looking at the display screen 304a.

  In this case, in particular, since the operation reaction force pattern in the direction of the detachment operation to the display object H2 is a pattern that rapidly increases, a large operating force is required to detach the cursor C that has entered the display object H2. The cursor C can be restrained to some extent in the display object H2, that is, the display object H2 can be set as a scroll range of the operation lever 310, and the display object H2 can be moved in an arbitrary direction. It can function effectively as a scroll area.

  In the above embodiment, since the reaction force application area includes the outer peripheral edge of the display object, the display data (coordinate data) indicating the outer peripheral edge of the display object is used as the coordinates of the outer peripheral edge of the reaction force application area. It can be used as data. In this case, the inner peripheral edge of the reaction force application region may not be present, that is, the entire display object may be the reaction force application region.

In the above-described embodiment, the in-vehicle device 30 further sends operation information on the display object of the operation lever 310 (information on the entry / extraction operation of the cursor C with respect to the display object or operation of the operation determination unit 314) to the mobile terminal 20. When receiving the operation information, the portable terminal 20 determines whether to change the display object on the display screen 205a of the display unit 205 based on the operation information, and changes the operation information. If it is determined, since the display content of the display screen 205a is switched, that is, the display content switching control is performed on the mobile terminal 20 side, the burden of the display switching control in the in-vehicle device 30 can be reduced.
In the above-described embodiment, since the operation reaction force pattern of the display object can be set and changed, the operation reaction force pattern can be set so as to match the user's preferred operation feeling, which is convenient.

The operation reaction force pattern is not limited to a mountain shape, a rectangle shape, an arc shape, or the like, but may be a concave shape or a jagged shape.
In the above embodiment, when the size of the display screen 304a of the display unit 304 is smaller than the size of the display screen 205a of the display unit 205 of the mobile terminal 20, the display screen 205a of the mobile device display unit 205 is reduced. A reduction rate calculation unit 212 for calculating a rate, display data (coordinate data) regarding the display position and display size of the display objects H1 and H2 to which the reaction force is applied, and the reaction force application region according to the reduction rate. A data changing unit 213 that changes data (coordinate data), and the data transmission control unit 213 sends the display data changed by the data changing unit 213 and the data of the reaction force application region to the in-vehicle device 30. It was set as the structure which transmits.

  In general, the screen size of the display unit 304 of the in-vehicle device 30 is larger than the screen size of the display unit 205 of the mobile terminal 20. In this case, the screen size of the display unit 205 of the mobile terminal 20 falls within the screen size of the display unit 304 of the in-vehicle device 30. However, when the screen size of the display unit 304 of the in-vehicle device 30 is smaller than the screen size of the display unit 205 of the mobile terminal 20, the display screen of the display unit 205 of the mobile terminal 20 is displayed on the display unit 304 of the in-vehicle device 30. Problems that do not fit on the display screen occur. In this regard, according to the above-described embodiment, when the screen size of the display unit 304 of the in-vehicle device 30 is smaller than the screen size of the display unit 205 of the mobile terminal 20, the reduction rate of the screen size is calculated. Since the display data regarding the display position and display size of the display objects H1 and H2 to which the reaction force is applied and the data of the reaction force application area are changed at the reduced rate, the screen size of the display unit 304 of the in-vehicle device 30 is the mobile terminal. Even when the screen size of the display unit 205 is smaller than 20, the display object can be displayed on the display unit 304 of the in-vehicle device 30 at the same display ratio as the display unit 205 of the mobile terminal 20.

  It should be noted that even if either the display screen size of the portable device display means or the display screen size of the control device display means is large, that is, when both display screen sizes are different, depending on the size ratio A change ratio of the display screen size of the display unit for portable devices is calculated (change ratio calculation means), and display data relating to the display position and display size of the display object to which the reaction force is applied according to the change ratio, and the You may make it change the data of a reaction force provision area | region (data change means).

  Moreover, it is good also as a structure which operate | moves as follows with the vehicle-mounted apparatus 30. FIG. That is, when the operation lever 310 is operated, the control unit 301 transmits the information to the mobile terminal 20, transmits display data and reaction force application area data from the mobile terminal 20 to the in-vehicle device 30, and the in-vehicle device. The display object is displayed and the reaction force is applied according to the data at 30. However, there may be a concern about the display delay of the display unit 304 or the reaction force application delay due to a delay in communication processing. In order to deal with this concern, the history of the operation content of the operation lever 310 in the in-vehicle device 30 and the display data and reaction force application area data transmitted from the mobile terminal 20 based on the operation content is sequentially stored, for example, the storage unit 307 (for history (History storage control means), and determines whether or not the operation content of the operation lever 310 is the same as the past operation content (operation history determination means). Based on the display data stored in the storage unit 307 by the unit 301, the display object is displayed on the display unit 304 of the in-vehicle device 30 without using the mobile terminal 20 (direct display control means), and an operation response to the display object is performed. The operation reaction force applying mechanism 311 is controlled (reaction force control means) to apply force.

In this way, when communication between the mobile terminal 20 and the in-vehicle device 30 is delayed, it is possible to prevent display delay and reaction force application delay on the in-vehicle device 30 as much as possible.
In addition, you may change and deform | transform the embodiment of this invention as follows. When the cursor C enters the display objects H1 and H2, the display form of the display objects H1 and H2 is changed (lighting, blinking, color change, etc.) to replace the cursor C. The display of C may be eliminated.

  In the above embodiment, for example, the user transmits the pressing operation of the operation lever 310 (operation of the operation determination unit 314) to the mobile terminal 20 as the operation information. However, for example, the cursor C enters the display object H2. Sometimes, this approach operation may be transmitted to the mobile terminal 20 as operation information. In this case, the mobile terminal 20 receives this cursor entry operation information for the display object H2, analyzes it, and executes a process corresponding thereto.

Further, the control device is not limited to the in-vehicle device 30 and may be a game machine, for example.
According to the embodiment of the present invention, when a mobile device and a control device are operated in cooperation to execute arbitrary application software of the mobile device, the display object of the reaction force application target is displayed on the display screen of the control device. When the selection operation is performed, an operation reaction force can be applied on the control device side, and the memory amount and control operation load on the control device side can be reduced.

  In the drawing, 10 is an operation reaction force imparting system for portable devices, 20 is a portable terminal (portable device), 201 is a control unit (portable device control means), 205 is a display unit (portable device display means), and 206 is drawing. 207 is a storage unit (portable device storage unit), 209 is a program providing unit (program providing unit), 210 is an extracting unit (extracting unit), 211 is a data transmission control unit ( (Data transmission control means), 212 is a reduction ratio calculation section (reduction ratio calculation means), 213 is a data change control section (data change means), 30 is an in-vehicle device (control equipment), and 301 is a control section (control equipment control means). , 304 is a display unit (control device display unit), 305 is a drawing control unit (control device drawing control unit), 310 is an operation lever (operator), and 311 is an operation reaction force application mechanism (operation reaction force application unit). ) 312 is a reaction force control unit (reaction force control unit), 313 is an operation direction detection mechanism, 314 is an operation determination unit, 315 is a display control unit (display control unit), and 316 is an operation information transmission unit (operation information transmission unit). Show.

Claims (9)

Various display objects are displayed based on the display data of the display device including the display device for portable device (205) and the display object to which the reaction force is applied and the display object not to be applied with the reaction force displayed on the display device for the portable device. A portable device (20) having a portable device drawing control means (206), and a portable device storage means (207) that stores reaction force application area data in the display object to which the reaction force is applied,
Control device display means (304), control device drawing control means (305) for displaying various display objects on the display screen of the control device display means, and an operator operated by the user (310) and an operation reaction force applying means (311) capable of applying an operation reaction force to the operation element when the operation element is operated, and a control apparatus capable of communicating with the portable device (30)
Extraction means (210) for extracting the reaction force application area based on the display object to which the reaction force is applied when the display screen of the portable device display means is switched;
Data transmission control means (211) for transmitting the display data and the reaction force application area data to the control device,
The control device further includes control device control means (301),
The control device control means includes:
The display data transmitted from the portable device is supplied to the control device drawing control means to display the display object on the control device display means, and the control device display means displays a display object instruction index. Display control means (315) capable of displaying a part and moving the indicator part on the display screen of the control device display means in accordance with the operation of the operator;
For portable equipment, comprising: a reaction force control means (312) for operating the operation reaction force application means when the indicator moves the reaction force application area of the display area of the display object. Operation reaction force application system.   The said reaction apparatus is a vehicle-mounted apparatus mounted in the vehicle, The operation reaction force provision system for portable devices of Claim 1 characterized by the above-mentioned.   The operation reaction force application system for mobile devices according to claim 1, wherein an operation reaction force pattern in the reaction force application region varies depending on a type of the display object.   4. The operation reaction force application system for portable devices according to claim 3, wherein the operation reaction force pattern can be changed by a user.   The said reaction force provision area | region contains the outer periphery part of the display area of the said display object, The operation reaction force provision system for portable devices in any one of Claim 1 to 4 characterized by the above-mentioned. The control device control means further includes operation information transmission means (316) for transmitting operation information of the operator to the portable device,
When the portable device receives the operation information, the portable device determines whether or not to change the display screen of the portable device display means based on the operation information. 6. The operation reaction force imparting system for a portable device according to claim 1, wherein the screen is switched. Furthermore, the portable device calculates a reduction ratio of the display screen size of the display device for portable device when the display screen size of the display device for control device is smaller than the display screen size of the display device for portable device. Reduction rate calculation means (212), and data change means (213) for changing display data regarding the display position and display size of the display object to which the reaction force is applied and the data of the reaction force application area in accordance with the reduction rate. )
7. The mobile phone according to claim 1, wherein the data transmission control unit transmits the display data changed by the data changing unit and the data of the reaction force application area to the control device. Equipment reaction force imparting system. Furthermore, when the display screen size of the portable device display means and the display screen size of the control device display means are different from each other, the portable device corresponds to the display screen size of the control device display means. Change ratio calculation means for calculating a change ratio of the display screen size of the display means for display, display data relating to the display position and display size of the display object to which the reaction force is applied, and the reaction force application area according to the change ratio Data changing means for changing the data of
7. The mobile phone according to claim 1, wherein the data transmission control unit transmits the display data changed by the data changing unit and the data of the reaction force application area to the control device. Equipment reaction force imparting system. The control device further includes history storage means, history storage control means, operation history determination means, and direct display control means,
The history control means sequentially stores the history of the operation content of the operator and display data transmitted from the portable device based on the operation content in the history storage means.
The operation history determination unit determines whether or not the operation content of the operator is the same as the past operation content stored in the history storage unit,
When the operation history determining means determines that the operation content of the operator is the same as the past operation content, the direct display control means is based on the display data stored in the history storage means. Display the display object on the control device display means of the control device without going through the portable device, and the reaction force control means based on the reaction force application area data stored in the history storage means 9. The operation reaction force application system for portable devices according to claim 1, wherein the operation reaction force application means is controlled.

Images