System Description

The teleoperation system described in the paper has been developed by International Submarine Engineering (ISE) Ltd. for the Canadian Space Agency (CSA) under STEAR contracts. The control system can be divided in two parts: the remote site consisting of dual 7dof manipulators and low-level control systems, and the local site which implements high-level control features and user interfaces (called the Operator Workstation).

The Operator Workstation software features extensive world modelling of robots and the environment. [Figure 1] The 3D model (shown in the right hand side of the screen) is used to help the operator visualize the environment; it can display either simulated information or feedback from the slave system. The model is also used by the control algorithms, e.g. by path planners for distance calculations.

The teleoperation control system we are describing is based on the NASREM control system architecture[1]. Consequently, it features extensive work task modelling (task knowledge base), necessary to support supervisory control modes. [Figure 1] Some example work tasks modelled in the knowledge base are shown in the hierarchical task list in the main user interface window (at the left hand side of the figure).

As mentioned earlier, the system supports a number of control modes, ranging from high level supervisory control to low-level manual control. During supervisory control operation, the operator issues high level commands or "task verbs" (such as acquire and place). The system decomposes these commands into lower-level symbolic commands (E-Moves), and then interprets them into numeric commands (by invoking path planner algorithms). The decomposition of the task verbs is performed based on the knowledge kept in the task knowledge base and the information from the world model.

[Figure 1] An example breakdown is shown in the hierarchical task list for the acquire task verb. This breakdown is done in two steps. The command is first broken into E-Move commands: position, approach, grapple, and extract. Secondly, each of these task verbs is interpreted (by invoking path planner algorithms) in order to obtain numeric commands suitable for execution by the lower level control system.

The robots can also be controlled in manual control mode via a hand controller device. The devices used in our system are a 6dof mouse, or a set of two 3dof joysticks. The input from the hand controller is transformed into Primitive level commands suitable for execution by the Robot Controller (effectively a stream of manipulator joint angles). Besides providing supervisory and manual control modes, our teleoperation system attempts to combine the best features of both of these control modes by using the notion of virtual fixtures. We can define the Virtual Fixture Control mode as an extension of the manual control mode, where virtual fixtures are placed in the environment based on the task knowledge typically related to supervisory control.

Virtual Fixtures

Virtual Fixtures can be defined as active, task-dependent software agents, whose purpose is to assist the operator during the preparation or execution of a task (task preparation refers to controlling a robot simulation, and will be used interchangeably with the task execution). The assistance provided by virtual fixtures may consist of force clues applied to a force-reflecting handcontroller, additional motion commands applied to the manipulator itself, or of imposed restrictions to the robot movements. When the teleoperation system features force-reflecting handcontrollers, virtual fixtures operate by providing force clues to the handcontrollers [4]. Otherwise, their operation is implemented using motion commands and movement restrictions, which is the case we will consider.

Table 1: Task Verb knowledge

Virtual Fixtures represent a task-dependent geometry overlayed onto the environment, and typically attached to an object in that environment. For example, a virtual fixture can be attached to a movable object to guide the robot gripper to the object's grapple point. As a geometry, the fixture imposes restrictions to the robot's movements, thus ensuring that the robot does not collide with the object. Further, the fixture applies additional motion commands to the robot to attract it to the grapple point, therefore simplifying the positioning process.

The key word in the virtual fixture definition is 'task dependent', and this notion has had a major influence on our implementation: we incorporated virtual fixtures into our task knowledge base. [Table 1] The task knowledge base consists of a number of task verbs, each of them containing information relevant to preparation or execution of the associated robotic work task. The virtual fixture data is added to the task verb knowledge base, and describes ways in which the system can assist the operator during the execution of the task using a virtual fixture control paradigm. Some responsibilities of the virtual fixtures in our system are:

* generating additional movements of the robots, e.g. homing motions

* restricting motion or DOFs of the robots

* setting low-level controller parameters, e.g. impedances

* providing cooperative control of the dual robots

Let us consider how fixtures assist the operator during the execution of the position or approach task (involves setting all 6 degrees of freedom of the robot end-effector). The operator normally provides the handcontroller input, while the fixture provides additional motion commands. [Figure 2] These motion commands are presented using the arrows (note that the figure only shows commands that relate to achieving the proper position of the end-effector in the workspace, while the commands that relate to achieving the proper orientation are left out).

Figure 2: Different clues provided by virtual fixtures

To avoid conflict with the operator's commands, the fixture considers only the degrees of freedom not specified by the operator at that moment. Thus, if the operator releases the handcontroller, the fixture will provide motion commands for all DOFs of interest. [Figure 3] As an example, the Virtual Fixture Control dialog box shows the handcontroller input from the operator (the dark bars), and the motion commands generated by the active virtual fixture (light bars). In this case, the operator sets the x coordinate and pitch plane angle, while the fixture controls other coordinates.

Figure 3: Interacting with the operator during task preparation

For some tasks it is beneficial to restrict the manipulator's motion (because of the interaction between robot and environment). For the approach task, we want to avoid collisions with the target, while allowing the operator to quickly approach the grapple position. For the extract task we want to simplify the extraction process, thus we restrict the degrees of freedom not involved in the process. [Figure 2] The restriction of motion or degrees of freedom is graphically shown by the double bars (only the restrictions related to position coordinates are shown).

In our system, the virtual fixtures are also responsible for setting low-level control system parameters. For example, we want a different type of robot control algorithm when the robot is in free space compared to when the robot is in contact with the environment. Based on the task knowledge, the virtual fixtures set the appropriate control parameters for the task being executed.

Finally, virtual fixtures are used to simplify the simultaneous control of dual robots. Suppose we want to perform a task where the coordinated motion of both robotic arms is required (e.g. to carry a heavy object). The virtual fixture knows that both robots need to be moved at the same time, and that additional constraints exist on the robots (they are coupled). In this case the virtual fixture uses the handcontroller input provided by the operator to generate setpoints for both robots, taking the task constraints into account. As a result, the operator is able to simultaneously control both robots using only one handcontroller device. After the coordinated task is finished (one of the robots ungrapples the object), the system goes back to operating only one robot at the time.

Operational Issues

The virtual fixture control mode can be used within two contexts: when providing an alternative for path planners during supervisory control, or when providing a task-dependent aid for manual control. Our system currently implements only the first case because of its reduced operational overhead. However, it is easy to use one instead of the other, and the major difference lies in the activation and deactivation of virtual fixtures.

When used as an alternative for path planners in supervisory control, the task that we want to execute is known in advance (it is specified by the operator). The appropriate fixture is placed in the environment (activated) when the operator starts the execution of the task using the virtual fixture control mode. When the task is ended (the system automatically performs this check based on sensory information and the postcondition knowledge of the task verb), the virtual fixture is removed or deactivated. The fixture can also be removed if the preparation of the task fails (based on the failure conditions).

Figure 5: Command generation by acquire fixture

When used as an aid for manual control, the work task is not known in advance (by the control system). Activation of virtual fixtures in this case is done reactively based on the task preconditions. Basically, one fixture is activated for each task whose preconditions (defined in the task knowledge base) are satisfied. When the task is ended or the failure conditions are satisfied, the fixture is deactivated.

Whether used as an alternative for path planners, or as an aid for manual control, virtual fixture control combines the supervisory and manual control approaches. When used as alternative for path planners, the fixtures make sure that the manually generated path plans are placed within the constraints of the work task. Also, during the generation process, task knowledge is used to simplify the operation (by placing appropriate virtual fixtures).

During manual control virtual fixtures can generate new high level commands (task verbs) to describe the actions that were specified by the operator during the work task. This task verb generation is performed when a virtual fixture is completed (i.e. when the task postconditions are satisfied). [Figure 4] Consider the acquire task consisting of the position, approach, grapple, and extract subtasks. Whenever a subtask is finished, the appropriate task verb is assigned to the path that was specified by the operator. [Figure 5] The positions of the end-effector at which these subtasks were completed are visually represented by black dots. When all four subtasks are complete, the acquire task verb is generated, and the previously generated subtasks are assigned to it.

System Evaluation

One of the benefits of virtual fixture control as compared to manual control is reduced operator workload. The fixture participates during the execution of the task, and offloads part of the work from the operator. For example, the operator could take care of setting the position coordinates, while the fixture could set the orientation. This would be an example of shared control of the manipulator.

In some cases, the fixture would be able to perform the task appropriately without any input from the operator. Thus the control responsibility would switch between the operator and the virtual fixture, which is an example of traded control.

Figure 6: Acquire fixture reducing operators' load

[Figure 6] Consider the real example of the acquire task performed using virtual fixture control. The operator is mostly involved in the first phase of the process (generating a collision free path). Once the end-effector is close to the fixture, a period of shared control exists where the operator sets the pitch plane angle (the redundant degree of freedom for the 7dof arm) if necessary, followed by autonomous fixture operation in finishing the task.

Figure 7: Performance measurements for the Position task

Virtual fixtures can also increase the speed and precision of robot operation. We conducted a simple test where several operators with skills ranging from novice to skilled performed a positioning task at their own pace. The task was first performed with virtual fixtures and then without them, and the duration of the operation was measured.

[Figure 7] The results indicate that the virtual fixtures significantly improved the operation speed for novice and intermediate operators. Further, all the operators considered virtual fixture control easier than manual control.

Concluions

The benefits of implementing virtual fixtures in our teleoperation system are multiple. With virtual fixture control we obtained a performance increase and reduced operator workload compared with ordinary manual control.

The virtual fixture control provided a scheme where manual control can be used as an alternative to path planners, while preserving the task information typically related to supervisory control. It also provided automatic command generation when used instead of manual control, thus establishing a closer relation between the supervisory and manual control paradigms.

Finally, we increased the reusability of paths generated with manual control. Since during the operation of the manipulator we obtained not just numeric paths, but their corresponding symbolic commands as well, we can easily check whether those path plans can be reused when that command is executed again.

Because of the initial modelling required for the virtual fixtures, they are appropriate for repetitive tasks, or those where speed and precision are essential. An example of future work can be an application of virtual fixtures to remote laparoscopic surgery.

---

References

1. J. S. Albus, H. G. McCain, R. Lumia. "NASA/NBS Standard Reference Model for Telerobot Control System Architecture", National Bureau of Standards Robot Systems Div., Dec 1986.
2. R. C. Arkin. "Three-dimensional Motor Schema Based Navigation", Proceedings of the NASA Conference on Space Telerobotics, vol. 1, pp 291-299, Pasadena, CA, Jan-Feb 1989.
3. L. B. Rosenberg, "The use of Virtual Fixtures to enhance telemanipulation with time delay", DSC-Vol. 49, Advances in Robotics, Mechatronics, and Haptic Interfaces, ASME 1993.
4. C. P. Sayers, R. P. Paul. "An Operator Interface for Teleprogramming Employing Synthetic Fixtures", Presence, Vol. 3, No. 4, pp. 309-320, Fall 1994.