Actin®  Version 5.3.0
Software for Robotics Simulation and Control
The Manipulation Director User Interface

Introduction

The manipulation director frontend is implemented in the ecManipulationDirectorGui library. The frontend can be broken down into the following groups: external interface, main window, director object library, script view, directions, and conditions. Additionally, it is possible to extend the frontend with custom pieces.

External Interface

The frontend is designed to be reusable, and a key piece of the reuse lies in the external interface. All clients of the frontend are required to implement the abstract interface EcManipulationDirectorGuiExternalInterface. This interface is the glue between the frontend and the backend. It provides the flexibility for the frontend to be used to control and monitor a backend that is running on a remote machine. The external interface includes three sets of pure virtual methods: basic interface methods, play control methods, and runtime modification control methods.

Basic Interface Methods

The following methods are included in the basic interface: “actorNames”, “numberOfActors”, “capabilities”, “availableLinks”, “registerDirectionEditorContainer”, and “registeredDirectionEditorContainers”.

“actorNames” Method

This method returns a string vector of actor names. This is generally an interface to the simulation to get a list of manipulator names that are contained in the simulation.

“numberOfActors” Method

This method returns an unsigned integer indicating the number of actors. This value should be equivalent to the size of the string vector returned from “actorNames”.

“capabilities” Method

This method returns an unsigned integer that includes a collection of bit flags. Each bit flag represents an optional capability that can be turned on or off in the frontend. The following is a list of capability bit flags: RuntimeInsertDirectionCapable, RuntimeRemoveDirectionCapable, RuntimeMoveDirectionCapable, RuntimeCueChangeCapable, RuntimeAppendSceneCapable, EditCastCapable, EditDirectorObjectLibraryCapable, EditScriptCapable, LoopCapable, LoadEditorCapable, and EditDirectionCapable. It is possible to change the capabilities dynamically.

Key Capabilities Bit Flag
Class Description
RuntimeInsertDirectionCapable As a general rule, it is not possible to make edits to the manipulation director while it is playing. However, if this capability bit flag is set, then it will be possible to insert directions into a scene at runtime. It will only be possible to insert the direction if the “insertDirection” method returns true.
RuntimeRemoveDirectionCapable As a general rule, it is not possible to make edits to the manipulation director while it is playing. However, if this capability bit flag is set, then it will be possible to remove directions from a scene at runtime. It will only be possible to remove the direction if the “removeDirection” method returns true.
RuntimeMoveDirectionCapable As a general rule, it is not possible to make edits to the manipulation director while it is playing. However, if this capability bit flag is set, then it will be possible to move, or rearrange, directions in a scene at runtime. It will only be possible to move the direction if the “moveDirection” method returns true.
RuntimeCueChangeDirectionCapable As a general rule, it is not possible to make edits to the manipulation director while it is playing. However, if this capability bit flag is set, then it will be possible to modify the cues in a scene at runtime. It will only be possible to modify the cues if the “directionCuesChanged” method returns true.
RuntimeAppendSceneCapable As a general rule, it is not possible to make edits to the manipulation director while it is playing. However, if this capability bit flag is set, then it will be possible to append a scene to the script at runtime. It will only be possible to append the scene cues if the “appendScene” method returns true.
EditCastCapable If this capability bit flag is set, then it will be possible to edit the cast. This involves specifying the role names and associating the actors with the role names. It could also involve changing the association between role-actor, adding or removing of a role
EditDirectorObjectLibraryCapable If this capability bit flag is set, then it will be possible to edit the director object library. This involves importing objects into the library, saving objects to the library, and rearranging the directory structure of the library.
EditScriptCapable If this capability bit flag is set, then it will be possible to edit the script. This involves adding or removing scenes from the script, as well as modifying scenes that are already in the script.
LoopCapable Occasionally, it is desirable to loop over a script. This starts the script from the beginning as soon as it successfully completes. If this capability bit flag is set, then the user can turn the looping on or off, and the user interface will display a loop count at runtime.
LoadEditorCapable The manipulation director frontend can be extended with custom additions. If this capability bit flag is set, then it will be possible to load these custom extensions. Refer Extensions for documentation regarding the extensions. It will only be possible to use the custom extensions if the “registerDirectionEditorContainer” method returns true.
EditDirectionCapable If this capability bit flag is set, then it will be possible to edit EcManipulationSceneDirections implementations inline.

“availableLinks” Method

This method returns a string vector of link names for a specified actor. This is generally an interface to the simulation to get a list of link names that are contained in the simulation.

“registerDirectionEditorContainer” Method

This method is used when the LoadEditorCapable bit flag is set (refer EditorCapabilities) to register extensions to the manipulation scene direction frontend. This method can return false if LoadEditorCapable bit flag is not set. Otherwise, it should register the new editor with the vector returned by “registeredDirectionEditorContainers” and return true.

“registeredDirectionEditorContainers” Method

This method returns a vector of all of the valid manipulation scene direction editors. This can include editors that are packages as part of the manipulation director frontend as well as custom extensions. The external interface can use this method to “lock down” the types of directions to allow.

Play Control Methods

The following methods are included for play control: “pause”, “play”, “resume”, and “stop”.

“pause” Method

This method requests the backend to pause execution. It will be called when the pause button is pressed, which can only happen when the manipulation director is currently executing.

“play” Method

This method requests the backend to start execution. It will be called when the manipulation director is currently stopped and the play button is pressed.

“resume” Method

This method requests the backend to resume execution after a pause. It will be called when the manipulation director is currently paused and the play button is pressed.

“stop” Method

This method requests the backend to stop execution. It will be called when the stop button is pressed, which can only happen when the manipulation director is currently executing or paused.

Runtime Modification Control Methods

The following methods are included for runtime modification control: “appendDataMapValues”, “appendScene”, “directionCuesChanged”, “insertDirection”, “moveDirection”, and “removeDirection”.

“appendDataMapValues” Method

This method requests the manipulation director backend to append data map values to the data map that gets passed into the backend during updates. This provides the ability for backend directions to respond to frontend user interactions. This is used, for example, to implement the manual pause direction.

“appendScene” Method

This method requests the manipulation director backend to append a scene to the script at runtime. It will only be called if the RuntimeAppendSceneCapable bit flag (refer EditorCapabilities) is set in the capabilities and the user attempts to append a scene to the script at runtime. If the scene is successfully appended, this method will return true. Otherwise, this method will return false.

“directionCuesChanged” Method

This method requests the manipulation director backend to modify cues in a scene at runtime. It will only be called if the RuntimeCueChangeDirectionCapable bit flag (refer EditorCapabilities) is set in the capabilities and the user attempts to modify cues at runtime. If the cues are successfully changed, this method will return true. Otherwise, this method will return false.

“insertDirection” Method

This method requests the manipulation director backend to insert a direction in a scene at runtime. It will only be called if the RuntimeInsertDirectionCapable bit flag (refer EditorCapabilities) is set in the capabilities and the user attempts to insert a direction in a scene at runtime. If the direction is successfully inserted, this method will return true. Otherwise, this method will return false.

“moveDirection” Method

This method requests the manipulation director backend to move a direction in a scene at runtime. It will only be called if the RuntimeMoveDirectionCapable bit flag (refer EditorCapabilities) is set in the capabilities and the user attempts to move a direction in a scene at runtime. If the direction is successfully moved, this method will return true. Otherwise, this method will return false.

“removeDirection” Method

This method requests the manipulation director backend to remove a direction from a scene at runtime. It will only be called if the RuntimeRemoveDirectionCapable bit flag (refer EditorCapabilities) is set in the capabilities and the user attempts to remove a direction from a scene at runtime. If the direction is successfully removed, this method will return true. Otherwise, this method will return false.

Main Window

Figure below shows the main window. Notice that there are four primary regions of the window: menu bar, director object library, play controls, and script view.

manipDirectorFramework_image14.png
Manipulation director frontend main window

Menu Bar

Figure below shows the file menu.

manipDirectorFramework_image15.png
File menu

New

This creates a new manipulation director file. f

Open

This opens a previously-saved manipulation director file. The default extension for manipulation director files is “.ecd”.

Open Path Planner

This opens a path planner file.

Save

This saves the current manipulation director to file. The default extension for manipulation director files is “.ecd”.

Save As

This saves the current manipulation director to file. The default extension for manipulation director files is “.ecd”.

Director Object Library

The director object library displays the contents of EcDirectorObjectDirectory. It provides the user interface to logically organize the library, and it allows many of the objects in the library to be dragged into the script view to compose scenes and scripts. Additional documentation for the director object library is provided in section Director Object Library.

Play Controls

The play controls include the following: play button, pause button, stop button, track check box, and gravity check box as shown in the Figure below.

playControls.png
Play Controls

Play Button

If the execution is currently stopped, pressing the play button will start the execution of the script displayed in the script view. Otherwise, pressing the play button will resume playing a paused script. This button will only be enabled of the script is not currently playing.

Pause Button

Pressing the pause button will pause the execution of the script. This button will only be enabled if the script is currently playing.

Stop Button

Pressing the stop button will stop the execution of the script. This button will be enabled if the script is either playing or paused.

Track Check Box

This check box indicates whether the script view will attempt to track the current position of a script that is playing. When the check box is checked, the script view will always try to vertically position the script so that the currently-executing directions in the script are centered. In order to make runtime changes to the script, it is often necessary to uncheck this box.

Gravity Check Box

This check box will turn gravity on and off for the script view. When gravity is off, the directions will raise to the level of their dependencies. While editing a script, it is often useful to turn gravity off in order to see the implicit and explicit dependencies between directions. At runtime gravity on provides more animation of the runtime process.

Script View

The script view displays the contents of EcManipulationScript (refer Manipulation Director Main Window). It visually displays the roles, actors, scenes, directions, and cues. Additionally, it allows many of the objects in the director object library (Manipulation Director Main Window) to be dragged into the script view to compose scenes and scripts. Refer to Script View for additional documentation.

Director Object Library

The director object library displays the contents of EcDirectorObjectDirectory (refer Manipulation Director Main Window). It contains reusable objects for composing scenes and scripts in the script view, and the objects can be organized in a hierarchical directory structure. Users interact with the director object library using context menus as well as drag-and-drop operations.

Context Menus

Context menus appear when we right-click in the director object library. There are three types of context menus displayed based on the region on which right click has happened.

1) Folder level context menu is displayed when we right click on a folder item in the tree.

2) Item level context menu is displayed when we right click on a leaf item in the tree

3) General folder level menu is displayed when we click outside of the tree.

Context Menus
WhereMenu Option Description
Background
New Folder This option prompts the user for the name of a new top-level folder to add to the director object library. The prompt will ensure that the new directory name is unique.
New Scene/ New Script/New Pose This option prompts the user for the name of a new scene/script/pose object to add to the director object library. The prompt will ensure that the new scene name is unique. The new scene will be empty; however, it can be inserted into the script view and modified.
Import This option prompts the user for the name of a file to import. Many different file formats are supported. The following file formats can be import into a directory: EcDirectorObjectDirectory. The following file formats will import a single object: EcDirectorObjectContainer, EcManipulationSequence, and EcWaypointSequence.
Import Directory This option prompts the user for the name of a directory to import. It will recursively search through the directory for all supported file formats and attempt to import each supported file. The following import formats are supported: EcDirectorObjectContainer, EcManipulationSequence, and EcWaypointSequence.
Export This option prompts the user for a file format to export and exports the entire director object library that is compatible with the selected format. The following export formats are supported: EcDirectorObjectDirectory, and Directory.
Folder Item
New Folder This option is similar to New Folder option from Background; however, the new folder is added as a child to the currently-selected folder item in the director object library.
New Scene/New Script/New Pose This option is similar to New Scene/script/pose option from Background; however, the new scene object is added as a child to the currently-selected folder item in the director object library.
ImportThis option is similar to Import option from Background; however, the imported objects are added as children to the currently-selected folder item in the director object library.
Import Directory This option is similar to Import Directory option from Background; however, the imported objects are added as children to the currently-selected folder item in the director object library.
Export This option is similar to Export option from Background; however, only the currently-selected folder item in the director object library will be exported.
Rename This option prompts the user for a new name for the currently-selected folder item. The prompt will ensure that the new directory name is unique.
Delete This option will remove the currently-selected folder item from the director object library.
Object Item
Import This option prompts the user for the name of a file to import. The following file formats are supported: EcDirectorObjectContainer, EcManipulationSequence, and EcWaypointSequence. All objects from the selected file will be imported. If any of the object names are same as existing ones, user would be prompted to either replace or cancel the import or use a new name
Export This option prompts the user for a file format to export and exports the currently-selected object item to the selected format. The following export formats are supported: EcDirectorObjectContainer, EcManipulationSequence, and EcWaypointSequence.
RenameThis option prompts the user for a new name for the currently-selected object item. The prompt will ensure that the new name is unique.
DeleteThis option will remove the currently-selected object item from the director object library.

Context Menus During Execution

It is not possible to modify the director object library during runtime; therefore, the context menus are simplified. Export option will be available during execution, it works as explained in the above table.

Drag-And-Drop Operations

The director object library supports drag-and-drop operations to organize the hierarchical directory structure and to simplify composing scenes and scripts in the script view.

Organize Directory Structure

Individual object items can be dragged to a different location by dropping into a folder item. A context menu will prompt the user whether to “Move” or “Copy” the object to the new destination. Similarly, individual folder can be dragged to a different location, and a context menu will prompt the user whether to “Move” or “Copy” the folder item to the new destination.

Compose Scenes and Scripts

A Scene is a ordered set of directions assigned to different roles that take part in the scene to accomplish a task. Thus, first step to compose a scene would be, to add the roles that will take part in the execution of the scene. Then we can drag and drop directions from the "Directions" tab into the role columns of the scene (refer Manipulation Director Main Window). The order in which these roles execute the directions are determined by their positions in the column and also by the cues. Typically all the roles will start simultaneously and execute the directions bottom up. If we want to synchronize the tasks performed by these roles, then we can add cues between the various blocks of the scene and control the order in which different roles perform different tasks/directions. Similarly, A Script is an ordered set of scenes that could be executed to accomplish a task with the available cast in the manipulation director and the simulation. We could drag and drop scene objects from the Scene Tab into the Script.

Drag-And-Drop Operations During Execution

During execution, the director object library cannot be modified; therefore, drag-and-drop operations to organize the directory structure are disabled. It is possible to drag direction objects into a scene if the RuntimeInsertDirectionCapable bit flag (refer EditorCapabilities) is set in the external interface. Similarly, it is possible to drag scene objects into the script header if the RuntimeAppendSceneCapable bit flag (refer EditorCapabilities) is set in the external interface.

Script View

The script view displays the contents of EcManipulationScript (Manipulation Director Main Window). It visually displays the roles, actors, scenes, directions, and cues. The script view has slightly different appearances, depending on the view mode. It can be subdivided into regions, and users interact with the different regions using context menus as well as drag-and-drop operations.

View Modes

When the manipulation director is not running, it is in edit mode. Otherwise, it is in execution mode. Furthermore, gravity can be turned on or off for the script view. This produces four view modes: gravity off edit mode, gravity on edit mode, gravity off execution mode, and gravity on execution mode.

Gravity Off Edit Mode

Figure below shows what the script view looks like in edit mode with gravity off. A cue is shown between the “Audience” role’s direction named “Start Clapping” and the “Snidely” role’s direction named “First Bow”. The cue line is the blue line between the directions. Notice that the dependencies between directions can clearly be seen when the gravity is off. Generally, gravity off is more useful while editing scenes and scripts.

manipDirectorFramework_image16.png
Script view in gravity off edit mode

Gravity On Edit Mode

Figure below shows what the script view looks like in edit mode with gravity on. It looks similar to the gravity off; however, all of the directions stack directly on top of each other when gravity is on. The cue line is more complex, and the dependencies between directions can not be seen unless hovering over a direction to display its cues. A direction which has a dependency on other block or has a cue will show the number of dependencies or cues it has below the name of the direction "Cues:<number of cues>". When we hover over such blocks which has Cues text, we can see the blue dotted lines showing the dependency.

manipDirectorFramework_image17.png
Script view in gravity on edit mode

Gravity Off Execution Mode

Figure below shows what the script view looks like in execution mode with gravity off. The same cue is shown between the “Audience” role’s direction named “Start Clapping” and the “Snidely” role’s direction named “First Bow”. Now, we can see which directions are currently executing by looking at the green background. All scenes and directions below the green area have already completed. All scenes and directions in the green area are currently executing. All scenes and directions above the green area are waiting to execute. Notice that there are progress bars on the scenes and directions that are executing. Also, all scenes and directions in or below the green area show elapsed times. The elapsed time for the overall script is shown in the scenes column header.

manipDirectorFramework_image18.png
Script view in gravity off execution mode

Gravity On Execution Mode

Figure below shows what the script view looks like in execution mode with gravity on. It looks similar to the gravity off; however, all of the directions stack directly on top of each other and a light green area is shown when gravity is on. The light green area indicates directions that are waiting for dependencies before they can execute. This means roles that have directions in the light green area are not currently active. This is very useful information during execution. The gravity also provides animation of the direction and scene transitions when directions and scenes complete. For this reason, gravity on is more useful while executing a script.

manipDirectorFramework_image19.png
Script view in gravity on execution mode

Script View Regions

The script view is subdivided into three regions: script header, scenes column, and role columns.

Script Header

The script header includes the header for the scenes column, as well as the header for each role column.

Scenes Column Header

This header will display a loop count if looping is enabled for the script, and it will display the total elapsed time of the script during execution.

Role Headers

Each role header will display the role name. If an actor is assigned to the role, then the name of the actor will be displayed below the role name. When hovering the mouse over a role header, the header will highlight.

Scenes Column

This column shows the scenes. When hovering the mouse over the scene, all directions associated with the scene are highlighted.

Role Columns

Each role column contains the directions for the role. When hovering the mouse over a direction, the direction’s cues will display, and the dependency directions attached to the cue lines will be highlighted.

Context Menus

Context menus appear when right-clicking in the scene view. The contents of the menu depend on if the mouse click is on a scene column header, role header, scene column, role column, or direction.

Scene Column Header Context Menu

The scene column header context menu provides the following options: “Add Role”, “Edit Cast”, “Toggle Loop”, and “Clear Script”.

Add Role

This option prompts the user to select a role name to add to the scene view. This will add a new role column and role column header.

Edit Cast

This option opens the edit cast dialog, which is shown in Figure below. This allows the user to add roles, rename roles, remove roles, and assign actors to the roles.

manipDirectorFramework_image20.png
Edit cast dialog

Toggle

This option toggles the loop flag on and off. The loop flag causes a script to restart from the beginning once it completes successfully. When the loop flag is on, the scene clumn header will display “[LOOP 1]”.

Clear Script

This option clears the script, by removing all the scenes from the script. Now we can add or configure the script the way we want from start.

Role Header Context Menu

The role header context menu provides the “Delete” option.

Delete

This option will remove the associated role, which will remove the role as an active participant in the script. Note, this does not remove the role from the actual Scene itself, it only removes it in the script.

Scene Column Context Menu

The scene column context menu provides the following options: “Edit”, “Save”, “Insert Scene Before”, “Insert Scene After”, “Merge Below”, and “Delete”.

Edit

This option opens up a dialog to edit the scene name.

Save

This option opens up a dialog to select a scene object inside the director object library to save this scene to.

Insert Scene Before

This option opens up a dialog to select a scene object from the director object library to insert into the script at a position below this scene.

Insert Scene After

This option opens up a dialog to select a scene object from the director object library to insert into the script at a position above this scene.

Merge Below

This option will combine this scene and the scene below it into a merged scene.

Delete

This option will remove this scene from the script.

Role Column Context Menu

The role column context menu provides the “Insert Direction” option.

Insert Direction

This option opens up a dialog to define a direction to insert. More documentation on defining directions can be found in Insert Direction Context Menu. If the user accepts the new direction by pressing the “OK” button in the dialog, the new direction will be inserted at the location of the context menu mouse click. For instance, if the context menu is launched between the second and third direction, then the third direction will be moved to fourth, and the new direction will become the third.

Direction Context Menu

The direction context menu provides the following options: “Edit”, “Edit Cues”, and “Delete”.

Edit

This option opens up a dialog to edit the direction.

Edit Cues

This option opens up the edit cues dialog, which is shown in Figure below. This allows the user to add, remove, and edit cues. Editing allows the user to specify the type for the cue (either SIMULTANEOUS or FOLLOWING), as well as the role and direction of the dependency.

manipDirectorFramework_image21.png
Edit cues dialog

Delete

This option will remove this direction from the scene.

Context Menus During Execution

There are fewer options for context menus during execution. Again, the contents of the menu depend on if the mouse click is on a role column or direction.

Role Column Context Menu

The role column context menu provides the “Insert Direction” option.

Insert Direction

This option opens up a dialog to define a direction to insert. More documentation on defining directions can be found in Manipulation Directions below. If the user accepts the new direction by pressing the “OK” button in the dialog, the new direction will be inserted at the location of the context menu mouse click.

Direction Context Menu

The direction context menu provides the following options: “Edit Cues” and “Delete”.

Edit Cues

This option opens up the edit cues dialog, which is shown in Figure Edit Cues. This allows the user to add, remove, and edit cues. Editing allows the user to specify the type for the cue (either SIMULTANEOUS or FOLLOWING), as well as the role and direction of the dependency.

Delete

This option will remove this direction from the scene.

Drag-And-Drop Operations

The scene view supports drag-and-drop operations to append scenes, insert directions, and move directions.

Append Scenes

Scenes can be appended to the script by simply dragging scene objects from the director object library to the script header of the scene view. This will work during execution as well if the RuntimeAppendSceneCapable bit flag (refer EditorCapabilities) is set in the external interface.

Insert Directions

Directions can be inserted into a scene by simply dragging either action objects or direction objects from the director object library to the appropriate role column of a scene in the script view. The drop location will determine the order for the inserted direction. This will work during execution as well if the RuntimeInsertSceneCapable bit flag (refer EditorCapabilities) is set in the external interface.

Move Directions

Directions can be reordered inside the role column of a scene by dragging it to the desired position. Cues can limit the available positions for the drop. This will work during execution as well if the RuntimeMoveDirectionCapable bit flag (refer EditorCapabilities) is set in the external interface.

Directions

Manipulation scene directions can be created and edited using the manipulation scene direction dialog. This dialog pulls in the list of available direction types from the “registeredDirectionEditorContainers” (refer Registered Editor Containers) method of the external interface. The following types are provided in the frontend: EcActionDirection, EcManipulationDirectionReference, EcAssignObjectRoleDirection, EcAttachToDirection, EcSetAttachmentDirection, EcCollisionExclusionDirection, EcConditionalDirection, EcToolOffsetDirection, EcManualPauseDirection, and EcSendSignalDirection. Additionally, it is possible to add frontend extensions for backend EcManipulationSceneDirection extensions.

EcManipulationDirectionReference

Figure below shows the manipulation scene direction dialog editing a “direction reference”, or EcManipulationDirectionReference (refer Manipulation Directions). Pressing the button the right of the action path prompts the user to select a direction object from the director object library.

manipDirectorFramework_image23.png
Manipulation scene direction dialog editing a \“direction reference\”

EcAssignObjectRoleDirection

Figure below shows the manipulation scene direction dialog editing an “assign object direction”, or EcAssignObjectRoleDirection (refer Manipulation Stage Directions). The user selects the role name of the object.

manipDirectorFramework_image24.png
Manipulation scene direction dialog editing an \“assign object direction\”

EcAttachToDirection

Figure below shows the manipulation scene direction dialog editing an “attach to direction”, or EcAttachToDirection (refer Manipulation Stage Directions). This directs one role to attach to another role, or it directs a role to detach from whatever role that it is currently attached to. The user first selects whether to “Attach” or “Detach”. If “Attach” is selected, then the user selects the “Base Role” and “Base Link”. The “Base Role” is the role that the directed role will attach to. The “Base Link” is the link on the “Base Role” where the attachment is made. If “Detach” is selected, then the user does not need to specify any additional parameters.

manipDirectorFramework_image25.png
Manipulation scene direction dialog editing an \“attach to direction\”

EcSetAttachmentDirection

Figure below shows the manipulation scene direction dialog editing a “set attachment direction”, or EcSetAttachmentDirection (refer Manipulation Stage Directions). This directs a role to add another role as an attachment to itself, or it directs a role to detach another role from itself. The user first selects whether to “Attach” or “Detach”. If “Attach” is selected, then the user selects the “Attached Role” and the “Base Link”. The “Attached Role” is the role that will be added as the attachment. The “Base Link” is the link on the directed role where the “Attached Role” will be connected. If “Detach” is selected, then the user selects the “Attached Role” to detach.

manipDirectorFramework_image26.png
Manipulation scene direction dialog editing a \“set attachment direction\”

EcCollisionExclusionDirection

Figure below shows the manipulation scene direction dialog editing a “collision exclusion direction”, or EcCollisionExclusionDirection (refer Manipulation Stage Directions). This is used to exclude or include collisions between specific links of two different roles. For instance, when connecting a gripper tool to a robot, certain links have to be excluded from collisions in order for the robot to get close enough to attach to the gripper. When detaching the gripper from the robot, the previously-excluded collisions between the robot and gripper should be included. The user selects whether to “Exclude” or “Include” collisions, and the user selects the links on the “Active Role” and the links to exclude or include on the other roles. If “Exclude” is selected, the selected links will be added to the collision exclusion map when the direction is executed (i.e. the links cannot collide in the simulation). If “Include” is selected, then the selected links will be removed from the collision exclusion map when the direction is executed (i.e. the links can collide in the simulation).

manipDirectorFramework_image27.png
Manipulation scene direction dialog editing a \“collision exclusion direction\”

EcConditionalDirection

Figure below shows the manipulation scene direction dialog editing a “conditional direction”, or EcConditionalDirection (refer Manipulation Direction). This is a complex direction that includes preconditions, persistent conditions, post conditions, and subdirections. The “Preconditions”, “Persistent Conditions”, and “Post Conditions” sections provide a context menu with the following options: “Add Element” and “Delete”. The “Add Element” option will open a dialog where the user can graphically design a conditional expression. Section 1.2.6 below provides documentation on editing conditional expressions. The “Delete” option will remove the currently-selected conditional expression. An existing conditional expression can be edited by double-clicking on it. Additionally, it is possible to attach a sequence of directions to the condition by double clicking on the exception box. This sequence of directions will execute if the condition fails. Similar to the condition sections, the “Directions” section has the same context menu. The “Add Element” option will open the manipulation scene direction dialog to define a subdirection. The “Delete” option will remove the currently-selected subdirection. It is also possible to attach a sequence of directions to execute if the subdirection fails.

manipDirectorFramework_image28.png
Manipulation scene direction dialog editing a \“conditional direction\”

EcToolOffsetDirection

Figure below shows the manipulation scene direction dialog editing a “tool offset direction”, or EcToolOffsetDirection (refer Manipulation Stage Directions). This is useful for adjusting the location of the end effector based on what tool is attached. For example, the robot might want the end effector to be located at the point of attachment to tools when tools are not attached to it. When attaching to the gripper, the robot might want the end effector to be between the gripper’s fingers. The user selects the tool offset index and specifies the “Tool Base Pose” and the “Tool Offset Pose”. The specified poses must be defined in the pose database. The actual tool offset will be the result of “(Tool Offset Pose) – (Tool Base Pose)”.

manipDirectorFramework_image29.png
Manipulation scene direction dialog editing a \“tool offset direction\”

EcManualPauseDirection

Figure below shows the manipulation scene direction dialog editing a “manual pause direction”, or EcManualPauseDirection (refer Manipulation Directions). There are no options that need to be set by the user for this direction.

manipDirectorFramework_image30.png
Manipulation scene direction dialog editing a \“manual pause direction\”

EcSendSignalDirection

Figure below shows the manipulation scene direction dialog editing a “send signal direction”, or EcSendSignalDirection (refer Manipulation Stage Directions). This can be used to inform other plugins when a specific point has been reached in the script. The user specifies a signal name and message, and clients running in other plugins can filter and determine the appropriate course of action based on these values.

manipDirectorFramework_image31.png
Manipulation scene direction dialog editing a “send signal direction”

Extensions

Figure below shows the interface for loading extensions into the manipulation scene direction dialog. These extensions can provide a frontend for EcManipulationSceneDirection backend extensions (Manipulation Director Extensions). When pressing the “Load Editor” button, the user will be prompted to load an “editor” file, which should have the extension “.ece”. Section 1.2.7 below documents how to create these frontend extensions.

manipDirectorFramework_image32.png
Loading extensions into the manipulation scene direction dialog

Conditions

Conditions are simple or complex expressions that evaluate to either “true” or “false”. The condition expression editor dialog is used to edit conditions. Figure below shows what this dialog looks like with an empty expression. Figure below shows the dialog with a complex expression. The complex expression is an example that might be used for a gripper role. The expression checks to see that “Milk Duds” is not currently attached to the gripper and that the gripper is attached to either the “Cyton” role or the “Robot” role. The expression can be edited using context menus and drag-and-drop operations.

manipDirectorFramework_image33.png
Condition expression editor dialog with an empty expression
manipDirectorFramework_image34.png
Condition expression editor dialog with a complex expression

Context Menus

Right-clicking on any node in the expression displays a context menu with the following options: “Edit”, “Insert NOT”, “Insert AND”, “Insert OR”, “Insert XOR”, “Delete”.

Edit

This option opens a dialog that allows the user to edit the condition. This allows the user to edit the following conditions: EcHasAttachmentCondition and EcIsAttachedToCondition.

EcHasAttachmentCondition

Figure below shows the condition edit dialog editing a “has attachment” condition, or EcHasAttachmentCondition (refer Direction Condition Implementations). The user can optionally specify the “Attached Role” and the “Base Link”.

manipDirectorFramework_image35.png
Condition edit dialog editing a “has attachment” condition

EcIsAttachedToCondition

Figure below shows the condition edit dialog editing a “is attached to” condition, or EcIsAttachedToCondition (refer Direction Condition Implementations). The user can optionally specify the “Base Role” and the “Base Link”.

manipDirectorFramework_image36.png
Condition edit dialog editing a “is attached to” condition

Insert NOT

This option inserts a NOT, or EcNotDirectionCondition (refer Direction Condition Implementations), node above the currently-selected node.

Insert AND

This option inserts a AND, or EcAndDirectionCondition (refer Direction Condition Implementations), node above the currently-selected node.

Insert OR

This option inserts a OR, or EcOrDirectionCondition (refer Direction Condition Implementations), node above the currently-selected node.

Insert XOR

This option inserts a XOR, or EcXorDirectionCondition (refer Direction Condition Implementations), node above the currently-selected node.

Delete

This option deletes the currently-selected node and all of its children.

Drag-And-Drop Operations

Any node in the expression can be dragged to any other node and dropped. When the node is dropped, a context menu appears with the following options: “Move”, “Copy”, and “Swap”.

Move

This option will replace the node on the drop side with the drag node. The original source for the drag node will be empty (stop sign).

Copy

This option will replace the node on the drop side with a copy of the drag node. The original drag node will not be modified.

Swap

This option will swap the drag node with the drop node.

Extensions

It is possible to extend the manipulation director with custom direction implementations. Any manipulation direction has two components, the backend and the front end. The backend component implements the actual aspects of the direction that acts on the manipulator or the robot to acheive the desired result. The front end for the direction provides an user interface that allows the user to specify the configurable parameters for the direction (For e.g, for an AttachToDirection, they could be whether to Attach or Detach). Adding extensions requires the following steps:

1) Backend implementation: Implement directions and directions state (refer Extensions)

2) Frontend implementation: implement the extensions or the interfaces provided for frontend, as explained later in the section

3) Build the extensions into a plugin

4) Use the plugin with the appropriate container classes.

Implement Directions

To implement a new direction, it requires a concrete implementation of EcManipulationSceneDirection abstract interface.

Implement Direction Widget

In order to edit the a new direction, it requires a concrete implementation of EcManipulationSceneDirectionWidget abstract interface.

Implement Extensions

Extensions can be added for the EcManipulationSceneDirectionEditor abstract interface. The interface requires the following methods to be implemented: “name” “description” “groupCode” “displayIcon” “createEditorWidget” and “directionToken” . Additionally, two static methods are also required: “classToken” and “creator”.

“name” Method

This method returns a name that is used in the manipulation scene direction dialog (refer Directions). This name should be unique.

“description” Method

This method returns a string that is used to describe the direction in detail. The string is currently used as a tooltip to display more information about the direction.

“groupCode” Method

This method returns a groupCode number that is used to differentiate the direction block inside a scene in different color.

“displayIcon” Method

This method returns a QIcon that is used to represent the direction symbolically in the director ui such as in the direction button or in the direction block in the scene. This icon should be unique.

“createEditorWidget” Method

This method returns a QWidget pointer that gets displayed in the manipulation scene direction dialog whenever the user selects the “name” type.

“directionToken” Method

This method returns a token that is used with the corresponding direction. This is used to associate the direction and direction editor with the same token. This name should be unique.

“classToken” Static Method

This method returns a EcToken for the extension class.

“creator” Static Method

This method constructs an instance of the extension class on the heap and returns a pointer to that instance as a EcXmlObject (base class) pointer.

Build Plugin

The extension implementations must be built into a plugin in order to be usable by the manipulation director backend. A plugin is essentially a shared library with the extension “.ecp” (or “.ecpd” for debug builds). The plugin requires two “C” functions to be defined: “getClassToken” and “getCreator”. Refer to Listing 1 above to see the declaration of these two functions.

“getClassToken” Function

Listing 6 below shows an example definition for the “getClassToken” function. The function should return a token for each extension class included in the plugin. This function will be called with an increasing index value until it returns an empty string. Each extension class should have a corresponding index, ad it must be consistent with “getCreator” (refer getCreator). When that index is passed into the function, it should call the associated extension’s “classToken” static method (refer "classToken") to get the appropriate return value.

Example definition of “getClassToken” function

const EcToken& getClassToken
(
EcU16 index
)
{
static const EcToken emptyString;
switch (index)
{
case 0:
{
return EcTimedWaitExampleDirectionEditor::classToken();
}
case 1:
{
return EcSendSignalExampleDirectionEditor::classToken();
}
default:
{
return emptyString;
}
}
}

“getCreator” Function

Listing 7 below shows an example definition for the “getCreator” function. The function should return a pointer to the static creator method for each class included in the plugin. This function will be called with an increasing index value until it returns a null pointer. Each extension class should have a corresponding index, and it should be consistent with “getClassToken” (refer getClassToken). When that index is passed into the function, it should return a pointer to the associated extension’s “creator” static method decsribed below.

Example definition of “getCreator” function

(
EcU16 index
)
{
switch (index)
{
case 0:
{
return EcTimedWaitExampleDirectionEditor::xmlObjectCreator;
}
case 1:
{
return EcSendSignalExampleDirectionEditor::xmlObjectCreator;
}
default:
{
return 0;
}
}
}

Use Plugin

In order to use extensions, the basename of the plugin must be specified in the “addLibraries” of the appropriate container class prior to setting the element of the container to the extension. Listing 8 below shows the example usage of a manipulation scene direction editor frontend plugin.

Example usage of a manipulation scene direction editor frontend

EcXmlStringVector addLibraries;
addLibraries.pushBack(EcXmlString("manipDirectorExtensionsGuiExample"));
EcManipulationSceneDirectionEditorContainer container;
container.setAddLibraries(addLibraries);
container.setElement(editor);

In the example, the plugin is named “manipDirectorExtensionsGuiExample.ecp”, and it includes “EcTimedWaitExampleDirectionEditor” as a direction editor extension. The corresponding XML for the above example is shown in Listing 9 below.

XML usage of a manipulation scene direction editor extension

1 <?xml version="1.0" encoding="UTF-8"?>
2 <manipulationSceneDirectionEditorContainer xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
3  xsi:schemaLocation="http://www.energid.com/namespace/ct timedWaitExampleDirection.ct.xsd" xmlns="http://www.energid.com/namespace/ct" version="4.1.0">
4  <timedWaitExampleDirectionEditor library="manipDirectorExtensionsGuiExample.ecp">
5  </timedWaitExampleDirectionEditor>
6 </manipulationSceneDirectionEditorContainer>

If the XML file is saved with the extension “.ece”, then it can be loaded by the manipulation scene direction dialog (refer Directions). Alternately, if the “.ece” file is placed in the directory named “externalDirectionEditor”, the manipulation scene direction dialog will load it automatically.

Created by Energid Technologies www.energid.com
Copyright © 2016 Energid. All trademarks mentioned in this document are property of their respective owners.