AR 3D Graphs Improve Comprehension over 2D

AR > 2D

Most mind maps are shown as 2D views of a graph with central nodes.   It is generally better for comprehension to see and interact with a 3D version of the same graph to understand the relations between its nodes.

In the paper Using augmented reality for visualizing complex graphs in three dimensions by D. Belcher, M. Billinghurst, S Hayes, and R. Stiles, published in the International Society for Mixed and Augmented Reality (ISMAR) 2003, the study results (link) showed that there is a significant advantage with reduced error for 3D views over 2D views of the same graph when trying to determine if nodes are connected (linked) to each other, which is a key capability when trying to comprehend a graph (or mind map).

The study shows that this improvement of 3D views and interaction with complex graphs also transfers to augmented reality, and not just desktop 3D.  In the graph from the paper, you can see significantly reduced node link comprehension error for 3D desktop and AR over the conventional 2D case, and this relationship holds as the number of nodes in the scene scales up.

 

Loci Memory Palace

We are proud to announce a mixed reality headset version of our Loci product, called Loci Memory Palace.

Get Loci for PC/MR on Windows Store
PC/MR-HMD

Loci lets you link your ideas in augmented or virtual reality, improving recall and links (associations) in your mind.  It is an interactive 3D mind map in mixed reality.  It runs on Windows Mixed Reality headsets with your Windows 10 PC.

In version 2018.8.1, we have added initial capability to import MindManager, Freemind/FreePlane, and GraphML files.   You can say “load graph” or press Ctrl-L on the keyboard and select .mmap, .mm, or .graphml files for import, along with .loci files now.  The GraphML importer supports nodes and edges, but does not yet support hyper-edges or ports.

It is available for download on the Windows Store from Microsoft here.

Click this link for info on the Windows 10 mixed reality home.

The headsets are available from HP, Lenovo, Dell, Samsung, Acer and others listed below, for an average of about $300 for a headset.  These mixed reality headsets use cameras to sense your surrounding area and position you in a room from the inside out, requiring much less setup than most VR headsets.

We provide a virtual reality location for your ideas and thoughts called the Memory Palace, so you can use the method of loci with immersive virtual reality mind maps.

Updated Voice and Gaze Commands

We updated the set of commands available in the mixed reality memory palace version to reflect capabilities available in that version, the new picture showing commands is below.

Usage in head set with controllers

Use your hand controllers to jump to locations in the Memory Palace, look at a location on the floor and your cursor will follow, then press forward on the controller joy stick to jump to that location.

Use the voice command “add node” to place nodes in the palace rooms.  Put in several nodes, one for each thought, then look look at the first node and say “link this”, then look at the next node so the cursor is on it, and say “to that”, and you will see a link appear between them.    Gaze at one of the nodes to put the cursor on it, and say “select” to move the node, and the links will update with the node.  You are building a mind map graph where the nodes have 3D locations in the memory palace.

Requirements

This version requires Windows 10 Creator’s build, a compatible Mixed Reality Head Mount Display and Controllers, and ideally a graphics card such as NVidia 1060.  Most commands use gaze and voice, but moving around in the MR scene is done with the Controllers similar to the way you move and rotate in the Cliff House starting scene.  Controllers are usually sold with Mixed Reality headsets.

Improvements

We added mind map and graph imports.  Also, with the Creator version of Windows 10, the Load and Save File Picker has better functionality, you can now change the name of the saved file, and both “load graph” and “save graph” come up more reliably with a file picker.  Evidently these bugs were fixed in the required separate Onedrive Windows File Picker service.

Known Issues

The first time you run Loci Memory Palace, it will start slowing as the operating system optimizes it for later loads, and you should run it until it shows a dialog box asking to allow voice input, use controller to select “yes” or say “yes”, then you will see the memory palace scene, and then you should exit to the Windows cliff house (HoloShell) by saying “go to start”, then look at the Loci window and say “select” to return back to Loci.

By doing this the Microsoft voice recognition service will work correctly, it seems to provide a reset for it to work after that point.  We are still tracking this one down, as it did not happen in earlier versions of Loci.  Also, it will be necessary to turn speech services on in your settings, see the fix here.

The HoloShell / Cortona reserved voice command word “select” is not always available while running our app for some reason.  It is the default Windows 10 way to select nodes, by gazing so the cursor is on the node and then saying “select” to pick or place a node.  Alternatively you can try our voice commands “pick node” and “place node” when moving nodes and select is not available.

Loci Voice and Gaze Commands

Loci Voice and Gaze Commands

You use Loci by saying commands and looking at nodes or locations in the scene.  Some commands are done only using your voice, and some include where you are looking, your center of gaze.

For Windows Mixed Reality, there are also a large set of useful commands here.  These include commands to operating system, like “select”, and commands to Cortona, like “Hey Cortona, take a picture”.  In this post we are focused on commands to the Loci application.

It is important to understand that you use your gaze to identify a node that you act on the node with your voice.    This is gaze-based selection combined with voice actions (commands).

Generally commands to change the graph are just voice, and commands to change nodes or links include you focusing your gaze onto a node, and seeing the donut cursor on that object.

Our demonstration video, six minutes long, provides examples of using voice and gaze commands: Demo Video

The image of Loci commands in this post is also included as the default image for every node created in Loci.  It appears when you start Loci as part of the default scene.

Basic movement of a node is done by looking at it to place the cursor, and saying “Select” which Windows HoloShell uses as a voice command like a left mouse button select, or using an air tap with your index finger.  Once you select a node, it locks onto your center of gaze and you can walk and move your head to place it.  If you have it where you want it, say select or tap again to place it.  It will be locked there for subsequent graph layout, since your chosen locations are very important for using Loci.

When starting  out, a very useful voice-only command is “Load Example” which will load an example mind map graph describing the Loci application.

The primary voice commands to use are:

“add node” creates a node at your center of gaze and allows you to use voice to name it and gaze to place it. If you look in mid air, it creates a new unattached node. If you look at an existing node, it adds a new node and conveniently also adds a link to it, and then Loci allows you to name and place the newly added node.

You can add links between nodes by focusing your gaze on a node and saying “Link This” then moving your gaze to another node and saying “To That” and then a link between them will appear.  You can delete a link by saying “Unlink This” while gazing at the first node, and then “From That” while gazing at the second node.

“delete node” deletes the node you are looking at (cursor follows gaze onto node).  This includes deleting links to that node.

“undo” allows you to undo a prior editing action, such as deleting a node

“select” is a Windows Mixed Reality standard voice command that is equivalent to a finger tap for selecting anything, in Loci it allows you to say “select” when gazing at a node and then place it with your gaze.  When you have the node in location, you say “select” again.

Say “name node” when you look at a node and Loci will ask you to provide a short name using your voice for dictation, and change the node name.

On the HoloLens version of Loci, saying “capture image” will use the camera to capture what you are looking at, create a node with your newly captured image shown below it, and allow you to place it.

A very useful set of commands are those for copy and paste.

You center your gaze on a node and say “Copy Node”.  This will take all the properties of a node and put them in the copy buffer in a text form.  Then you can look at a place in open space and say “Paste Node” and a copy will appear one meter in front of where you were looking.  You can also focus your gaze on an existing node, and say the same command “Paste Node” and it will update that node with the properties of the node you have previously copied (Color, scale, name, etc.).

Sometimes you only want to paste certain properties.  So you do the command “Copy Node” as usual, and then focus your gaze (cursor) on another node and say paste with a property name, such as “Paste Name” or “Paste Image” or “Paste Color”.

Also see our post on proper Windows 10 settings for voice commands here.

 

Graphs

Loci is a general graph editor that supports mind map graphs. Graphs consist of nodes and edges. We call nodes with locations loci, and edges associating loci we call links.

node + location in real world = locus (loci is plural)
edge = relation = association = link

Many items in life can be understood using graphs. From molecules to web page links to friends of your friends, you can think of things as the nodes, and relations between them as the links.

Graphs are used in mathematics, physics, and informatics as underlying representations for solving problems.  Graph data structures are used for 3D games to manage the application of transformations, and Loci itself uses such a scene graph to draw all the things it shows.

Most mind map software requires a central concept node, and links to other nodes, with no node being unlinked, and all being directly, or indirectly connected to the central node.  Loci does not require that.   Nodes can be solitary, or multiply connected.   In this way we support the analysis process from end to end.   You can make unlinked notes for things as you observe them, or your ideas as you come upon them, then orient your thoughts by linking them together to make sense of them.