Controls

• Concept and Use Cases
• Use Controls
• Create Controls
• Edit Controls
• Expand Controls
• Augmented Reality Controls (AR Controls)
• List Control - Snapping Mechanism
• Breadcrumb - StateMachine Workflow

Concept and Use Cases

Control Concept

Controls are template scenes created for later reuse via control nodes. They are meant to be interactive. To make them interactive they can be filled with Behaviors. A control collects a part of the scene graph and Behaviors into a single entity, which can be instantiated in another scene. All the controls are written in a special section which belongs to the asset library. When the solution is loaded from the asset, the content of the value of the control properties from the control node is set on the associated Behaviors in the control. Also the anchor nodes inside the user scene are added to the associated nodes in the control.

Created controls will include Behaviors with the observation that the Behaviors will be included only if they are bound to a node from the created control.

Main differences of controls compared to regular scenes:

Each scene node is uniquely identified by its parent using the type and the name. To identify a node globally its path to the scene should be used (for example /Scene:MyScene/Group:RootNode/Mesh:MyMesh).

The Scene Tree panel of a scene using a control will treat it as a single entity with all available anchor nodes as children. It is not possible to change the content of the control node, only the content of the anchors can be changed.

The following impact has to be considered: Cross-references (nodes or behaviors) to different scenes may occur. For details see Cross-Reference Implications.

Control Use Cases

Use Controls

Use Controls

Controls are displayed in Toolbox panel from where they can be dropped into scenes which will result in control nodes.

A "Show Controls as Big Icons/List" button is available in the upper right corner. If this button is used, a different presentation mode - as icons instead of a text list of items - will be made available (see the image below).

The Scene Tree panel of a scene lists a control node as a single entity with all the anchors defined as children.

New content (other nodes) can be added from the Import or Toolbox panels to anchors by dropping a new node over an anchor and editing it in the Properties panel as usual.

Control Properties

By using the context menu the "Configure Public Control Properties" panel will get opened. This can be done by selecting the "Control Properties..." operation from the context menu.

In this way, a new dialog "Configure Public Control Properties" opens where the user can choose which properties of the widget should be available (i.e. public) in the control.

If necessary, the user can use the fields available under the "Name" and "Tooltip" sections to introduce any useful piece of information. The values filled by the user in the "Configure Public Control Properties" dialog will be visible in the "Properties" panel.

All the name of the properties have to be distinct/different. If not, a "Property name must be unique." warning message will be displayed.

If all the names are different, the properties which were selected as "public" will be displayed as public properties in the Properties panel. It is possible to remove any of these properties by using the "X" button.

If the control with public properties is used in a scene, those properties will be visible in the Properties panel, too.

When the control is used in a scene, any of the public properties can be set or modified but none of the properties can be removed.

The display order of public properties in the property panel can be set in the [ Public Control Property Configuration ] dialog. You can change the display order by clicking the button to the left of each property name displayed in the Public properties, or by dragging and dropping each property.

Control Updates

When a control is being edited, the changes are reflected in all the scenes using that control.

---

Create Controls

Save a Node As Control

Save nodes as control using:

In the "Add New Template" dialog, a location within the solution and a name for the new control is defined.

The new control will be available in the

Open the control for editing with a double click.

Create an Empty Control

In Solution Explorer, use the context menu "Add New Item" to create a new control.

 

Edit Controls

Edit Controls

Controls are listed in Toolbox-Controls panel and Solution Explorer panel. By double clicking on a control, it will be opened for editing just like a normal scene.

Editing controls allows to:

It is possible to set a description for the control properties. This description will be shown as tooltip in the properties panel for the associated property of the the control node.

Create Anchors

The Toolbox panel provides an "Anchor" node in addition to regular scene toolbox items but no cameras, when editing controls. An anchor will later be used by a scene to attach further content to the control.

Add an anchor dragging it from the Toolbox and drop it on a desired node part of the control. The anchor will appear as a new child of the control in the Scene Tree panel, named after the parent node on which it is dropped:

The node an anchor is assigned to can be modified in the Properties panel of the anchor.

Further, all anchors of the control are also listed in the Properties panel of the control.

The control functionality provides support to add default content to anchors. When a control is instantiated in a scene, in the anchors where the "Default content" is specified, a control node of that type will be instantiated and added as child to the anchor node itself.

A specific option is available in the context menu of a control instance which has an anchor with default content. The name of this option is "Reset Anchor". By using this option it is possible to reset anchors to the default content.

Delete Anchors

Delete an anchor in the Scene Tree panel via context menu or in the Properties panel of the control by using the "X" button on the left side of the anchor:

 

Expand Controls

Expand Controls

If a control node needs to be customized for a specific set of scenes, it might be useful to decompose an existing control to create a new, customized version of it.

For this, use the context menu option "Expand Control Node" on a selected control.

In this case, the control node will lose the link to the original control source and will become a usual node; any modification of the control will not affect it anymore.

 

Augmented Reality Controls (AR Controls)

1. Introduction

This section describes CGI Studio controls to enable augmented reality applications with CGIStudio, which were designed for generic navigation overlay use case in the automotive area. Typically in video see-through AR, a real world video feed is enhanced with graphical elements.

2. Use Case: Navigation Overlay

Major elements for this use case are:

Detailed descriptions can be found in Predefined Controls and Predefined Behaviors.

3. Prerequisites

3.1. Cmake flags

Enable and disabling mixed reality and streaming features:

3.2. GStreamer

GStreamer is the default implementation of the VideoStreamer interface. It is used to decode the video source and provides the data to the VideoStream Behavior. If you want to use GStreamer, a working installation of GStreamer is required. The GStreamer version provided in the CGI Studio 3PSW is GStreamer version 1.16.1. Successfull installation of GStreamer modules can be verified with the console commmand gst-inspect-1.0.
Note that, under Windows you might need to add GStreamer to the environment variables. Instructions can be found here: https://gstreamer.freedesktop.org/documentation/installing/on-windows.html

Please also note that, the Android Evaluation Package comes with an installer and by default, the CGI Player is built with streaming enabled. The recommendation is to check the "Install GStreamer" option in the provided installer, and do a "Repair" installation regardless of whether GStreamer is installed or not, because of some dependencies.

If in any case the CGI Player still looks for a missing dll, check that the GStreamer is set in the PATH environment variable, or copy the GStreamer dlls to the same location as the CGI Player executable.

For more general information, please visit the official GStreamer website: https://gstreamer.freedesktop.org/.

3.3. How to build the CGI Player using GStreamer and Microsoft Visual Studio

Finally, you can start the Player application.

4. Interfaces

4.1. Events

4.2. Messages

5. Known Issues and Limitations

5.1. Streaming

5.1.1 Video Stream

5.1.2 GStreamer

5.2. Mixed Reality

5.2.1. PoseTracking

5.2.2. Points of Interest

List Control - Snapping Mechanism

Snap List Item Extension

The Snap List Item Extension Behavior is a Control > Scroll Behavior that allows to configure a snapping mechanism for a List Control. To use this snapping mechanism, drag and drop a Snap List Item Extension behavior onto a List control in your Scene Editor or Scene Tree panel.

The Snap List Item Extension behavior considers the following use cases:

• When a new item is focused, the list snaps to this item at the aligned snapping position of this item with a snapping animation.
• When the list is scrolled by direct touch, scroll of the list or by external scroll via scrollbar, the list will snap to the item with the smallest distance to the aligned snapping position.
• When the list is scrolled by a next/previous page scrollable event the list will snap to the item with the smallest distance for that new page to the aligned snapping position.
• When the list is scrolled by a next/previous item scrollable event, the list will snap to the newly requested item to the aligned snapping position.
• When a list is flicked by a touch gesture, the final target position will be adjusted in a way that the list snaps to the item with the smallest distance at the flick target position that is determined by the flick speed.

This snapping mechanism supports

• static and dynamic lists
• vertical and horizontal lists
• left-to-right and right-to-left direction
• the wraparound feature for static lists
• restoring the last focused index
• disabled items (Note: not supported for dynamic lists)

Wraparound is only available for static lists with an accumulated complete item size that is larger than the arrange area of the list.

Configuration

The Snap List Item Extension behavior can be applied to a list control via drag and drop from the toolbox.

In case you are using a custom ListControl, it is recommended to configure the ListBehavior of the ListControl as the ListBehavior of the SnapListItemExtension behavior. Please see Configure ListBehavior in Custom List for more information.

The properties List Alignment and Item Alignment define the snapping position. The snapping position can be refined by the List Alignment Offset and Item Alignment Offset properties. The properties List Alignment Offset In Percent and Item Alignment Offset In Percent define if the unit of the offset is in percent or in pixel.

List/Item Alignment

 These properties can be set to 

• Begin
• Center
• End

The base alignment for item/list is defined as follows:

List/Item Alignment Offset

The Alignment Offset properties allow to refine the snapping position.

The following rules apply to the alignment offset properties:

• For begin and center alignment the offset is directed towards the end of the arrange area.
• For end alignment the offset is directed toward the begin of the arrange area.
• In case of percental offset configuration, the offset value has to be between 0 and 1 and will be multiplied with the alignment area size to calculate the absolute offset.

Aligning the List and the Snapped Item

The configuration in the properties describe an alignment point for the list and an alignment point for the snapped item. The alignment point of the list and the alignment point of the item shall be aligned at the same visual position for a snapped item.

For a normal ListBehavior instance the item alignment starts with the first item presented on top of the list arrange area and stops with the last item at the bottom list arrange area:

So, the accumulated size of all items represents the complete size of the virtual area presented by the list that can be accessed via scrollbar and scrollable events.

Due to the static item snap position the list

has to present additional empty space before the first item as a begin offset and also after the last item as an end offset:

So, the normal list without snapping extension presents an undecorated complete size of its items. But the snapping extension has to add the begin and end offset to this undecorated complete size to present the decorated complete size as complete size of the scrollable interface:

This begin offset also affects the position of the item. So, the first item is virtually positioned at the virtual position of the begin offset. The final aligned snap position of an item has to reflect the begin offset, the accumulated size of all predecessor items, the item alignment offset and the list alignment offset.

Find the Matching Snap Item

Whenever the position of the items in the list is changed (e.g. when scrolling or flicking or the like) the snapping extension will determine the item to snap to. The item, whose alignment position has the least distance to the list alignment position, will be the matching snap item.

Configure ListBehavior in Custom List

The Snap List Item Extension's ListBehavior can be configured in the custom List Control.

Open your custom list control with a double click or create your custom list control by dragging the predefined list control from the Toolbox into the Controls folder of your solution in the Solutions Explorer.

The CustomList looks like this in the Scene Tree:

Drag and drop a Snap List Item Extension behavior to the CustomList:

Drag the ListBehavior from the CustomList's Behaviors down to the SnapListItemBehavior's ListBehavior:

Now you can see that the ListBehavior property has received a new value:

Snap List Item Configuration

The alignment position of a list's items is configured in the Snap List Item Extension behavior with the following properties:

• ItemAlignment
• ItemAlignmentOffset
• ItemAlignmentOffsetInPercent

In case certain items should be configured with a different alignment position, you can apply a Snap List Item Configuration behavior to those items that should be handled differently.

Drag and drop this behavior from the Toolbox onto the desired list item to be able to configure a different alignment position for this item.

Breadcrumb - StateMachine Workflow

Breadcrumb is not a standalone control, it does not trigger internally any scene transitions so it should be always connected with external state machine. For that purpose, breadcrumb property State Machine Node should be set to a node that contains a State Machine behavior.

Data flow for SetPath:

Data flow for BackToPrevious action:

Data flow for BackToElement action:

State Machine Example

1) Create a new state machine in Solution explorer tab as explained in the State Machine Basic Operation documentation.

2) Open the newly created state machine.

3) Create an Initial State and a State and connect them, it will be the main state that manages other states.

4) Create the Initial State and a State again but this time inside the MainState.

5) Create several sample scenes. On the BreadcrumbHub scene add a Breadcrumb control.

6) In the state machine editor and select the state called "Level0". Set the actions to be called on entry and on exit. At this moment, the path will be set. That scene is first so we only send the name without any separators via Breadcrumb action. Set BreadcrumbNode on BreadcrumbHub scene. Set active / deactivate transition request for selected scene.

7) Create the next states similarly.

8) Fill each state with actions that are to be called. Below is an example of how to pass nested path with several levels.

9) After completing all actions for entry and exit in states, we can create transitions when the user clicks on the breadcrumb element. It is a dragging line from the main state to the state we want go back. It should be remembered that we will never go to the final states so Level 1.1 and Level 2.1 will be omitted. On every transition we create a condition. Behavior sends the path from the root to the selected element using ValueChangedEvent.

Create similar transition for Level1 and Level2. Don't forget to set the Condition Value to Level1 and respectively Level2.

10) [optional] After all transitions with the paths from breadcrumb are defined, you can create transitions that can be used to change the active scene to the next scene by clicking the button on the active scene.

11) The state machine is on the BreadcrumbHub scene so if we want to receive click events in our state machine, we had to add event bridge to other scenes and set target node property to node with state machine.

Sample Solution