Formline

Space 3D Manipulation UX Optimization

Status: Decision-complete implementation plan

Summary

Replace Space view’s mode-heavy, gesture-competing controls with platform-native camera navigation and a contextual planning transform tool. The result should feel calm and direct on iPad, precise with Apple Pencil, and native with a Mac trackpad or mouse.

The current implementation has several sources of friction:

• A single tap can select, focus, or recenter the camera, making outcomes unpredictable.
• Orbit, Pan, and Top are exposed as persistent modes even though standard gestures can handle camera movement directly.
• Object editing and camera gestures compete in one pan handler, while all recognizers allow simultaneous recognition.
• Furniture resize and height handles scale in world space, so their apparent size and usability vary with camera distance.
• Move and footprint resize do not consistently use the global snapping system.
• Drag previews can rebuild the complete RealityKit anchor, creating avoidable work on every frame.
• Essential navigation controls become difficult to discover in compact portrait layouts.

Interaction and UI

Navigation

• Replace the Orbit/Pan/Top mode group with a 3D / Top view switch and remove persistent Pan mode.
• Make Fit preserve the active view: fit the perspective camera in 3D and the orthographic camera in Top.
• Use Focus as a contextual action for a selected object rather than changing the camera on ordinary selection.
• A single tap selects only. A double tap on an object selects and focuses it. A double tap on empty space, or the Fit control, frames the whole space.
• Keep camera navigation available without selecting a mode:
  • iPad touch: one-finger drag on empty canvas orbits, two-finger drag pans, pinch zooms, and two-finger rotation yaws the camera.
  • Apple Pencil: selects and manipulates objects; it must not accidentally orbit when a transform target is acquired.
  • Mac trackpad: two-finger drag orbits, Shift + two-finger drag pans, and pinch zooms.
  • Mac mouse: secondary-button drag orbits, middle-button drag pans, and the wheel zooms. Primary-button input selects and transforms.
• Keep Option-drag camera elevation as an advanced Mac shortcut, but remove camera raise/lower chevrons from the primary UI.
• On iPad, place a compact 3D / Top / Fit navigation island within the unobscured canvas safe area so it remains visible in portrait. On Mac, keep the equivalent controls in the top bar.

Contextual transform tool

• When a supported object is selected, show a compact contextual strip with Move, Rotate, Size, and Focus. Only the active transform’s handles are visible.
• Use quiet, neutral styling consistent with Formline. Do not introduce colored XYZ axes or a desktop game-editor gizmo.
• Keep every interactive handle screen-space sized with a minimum 44-point hit target. Scale its rendered geometry to maintain a stable apparent size as the camera moves.
• Show a nearby tabular value readout during manipulation. Tapping a value opens numeric entry for exact edits.
• Move:
  • Drag the selected body or move handle across the floor plane.
  • Authored shapes additionally expose a vertical elevation stem.
  • Placeable objects remain upright and grounded.
• Rotate:
  • Show a screen-space-stable yaw ring around the object.
  • Drag continuously when Snap is off.
  • When Snap is on, use 15-degree increments with stronger 45-degree detents.
• Size:
  • Show local width, depth, and height face handles.
  • Keep the opposite face fixed while resizing width or depth.
  • Keep the base fixed while resizing height.
  • Apply catalog minimum/maximum dimensions where defined.
  • Scale authored mesh vertices about the fixed opposite face; do not add a persistent scale transform.
• Apply the shared Snap resolver to all eligible move, rotate, and size operations. Snap settings must affect 3D manipulation consistently with the rest of the app.

Scope

• Enable the full planning transform for furniture, real-world placeholders, Build-linked placements, Volumes, and authored shapes.
• Limit transforms to floor-plane position, yaw rotation, width, depth, and height. Authored shapes may also change elevation.
• Keep rooms, walls, openings, zones, and labels editable through Top view or their inspectors; do not add freeform 3D transforms to semantic construction elements.
• Do not add pitch, roll, or unconstrained floating transforms.

Implementation

State and interfaces

• Replace camera-tool state with a two-value SpaceViewMode representing perspective and Top.
• Add explicit transform state:
  • SpaceTransformMode: move, rotate, or size.
  • SpaceTransformCapabilities: the operations and dimensions supported by the selected object.
  • SpaceTransformHandle: body, elevation, yaw, width faces, depth faces, and height face.
  • SpaceTransformSession: immutable start state plus the active handle, drag constraint, snapping context, and preview values.
• Consolidate the scattered mesh, height, footprint, body-move, and camera callbacks behind one transform lifecycle: begin, change, end, and cancel.
• Add whole-object width, depth, and height fields to the authored-shape numeric inspector. Commit authored-shape scaling by updating local vertices and normalizing the object bounds; do not introduce persisted scale, pitch, or roll fields.
• Route all commits through the existing undoable document mutation path. A cancelled or interrupted gesture restores the session’s start state without adding an undo entry.

Gesture coordination

• Replace the unconditional simultaneous-recognition policy with explicit recognizer dependencies and input ownership.
• Resolve a transform target before camera navigation begins. Once acquired, that pointer or Pencil sequence belongs to the transform until it ends or cancels.
• Give two-finger navigation precedence over one-finger object/camera gestures and cancel the lower-priority preview cleanly if a second touch is added.
• Make single-tap selection wait for double-tap focus recognition to fail.
• Add pointer and Apple Pencil hover previews for selectable objects and handles where the platform supports them.
• Keep gesture routing testable outside the RealityKit view by expressing the input, hit-test result, current view, and selected transform mode as explicit decisions.

Rendering and performance

• Keep the RealityKit root anchor, camera, lights, floor, and unaffected entities stable during camera and transform gestures.
• Update the active entity directly for move, rotate, and size previews instead of rebuilding the complete scene anchor.
• For authored-mesh resizing, regenerate and replace only the active mesh entity. Reuse cached materials and collision shapes where valid.
• Coalesce preview updates to display cadence, then perform one authoritative document commit and scene reconciliation when the gesture ends.
• Throttle expensive fit/diagnostic work during previews and run authoritative validation at commit.
• Preserve the in-progress adaptive top-bar and wall-visibility work as the implementation baseline.
• Put new transform UI and gesture-routing types in focused Space files rather than expanding SpaceWorkspaceView.swift further. Update docs/codebase-index.md if source files are added or responsibilities move.

Education, Accessibility, and Motion

• Show a one-time dismissible navigation hint tailored to Touch/Pencil or Trackpad/Mouse. Persist dismissal and retire the hint automatically after the user successfully orbits, pans, and zooms.
• Update the Help guide with the new platform-specific navigation gestures, selection/focus behavior, transform modes, snapping, numeric entry, and Top-view alternatives.
• Give every control and handle a useful accessibility label, current value, and action hint. Preserve Top view and numeric inspectors as accessible alternatives to spatial dragging.
• Announce snap points, rotation detents, dimension limits, and rejected moves without requiring vision.
• Render transform previews immediately. Use restrained 150-220 ms transitions only for contextual HUD appearance and state changes.
• Respect Reduce Motion by removing spatial HUD transitions and nonessential camera animation.
• Do not require continuous haptics; use sparse feedback only for meaningful detents or limits on supported hardware.

Test Plan and Acceptance Criteria

Automated tests

• Unit-test gesture ownership, recognizer failure order, second-touch cancellation, single-versus-double tap behavior, selection stability, and Fit preserving 3D or Top view.
• Unit-test transform math for floor-plane movement, authored-shape elevation, yaw rotation, 15/45-degree snapping, fixed-face resizing, fixed-base height resizing, catalog clamps, and mesh vertex normalization.
• Cover degenerate mesh bounds, minimum dimensions, linked-object status, preserved face colors/materials, undo/redo, cancellation, and document reload.
• Assert that camera movement and whole-object previews do not remove the root anchor. Assert that authored-mesh resizing replaces only the active mesh entity.
• Add UI coverage for portrait, landscape, and narrow Mac layouts; contextual modes; 44-point targets; live and numeric readouts; 3D / Top / Fit; Help content; and Reduce Motion.
• Add Mac-specific routing tests for trackpad modifiers, mouse buttons, wheel zoom, and primary-button transform isolation.

Device validation

• Complete hands-on QA on a physical M-series iPad with touch and Apple Pencil, plus a Mac with both trackpad and mouse.
• In a typical furnished room, maintain 60 fps with p95 frame time below 16.7 ms during camera and object manipulation.
• Keep input-to-visual response below 50 ms for transform previews.
• Confirm that no complete anchor rebuild occurs during camera movement or whole-object preview.
• In the supported 64-shape stress scene, update only the active mesh while resizing and allow no interaction hitch longer than 50 ms.
• Verify that gestures never leave a partial preview, duplicate undo entry, unexpected camera jump, or changed selection after cancellation.

Assumptions and Defaults

• Space remains a planning and spatial-understanding surface, while Top remains the primary precision-editing surface for construction geometry.
• The full transform tool applies to placeable objects and authored shapes only; semantic construction elements keep their existing editing paths.
• Existing persisted document models remain compatible. Authored-shape size changes are baked into vertices rather than stored as a new scale property.
• Existing uncommitted adaptive top-bar and wall-visibility changes are preserved and treated as the baseline.
• The interaction design follows Apple’s gesture, Apple Pencil, hover, gesture-coordination, and 44-point target guidance, with platform-native CAD navigation conventions used where Apple does not prescribe domain-specific behavior.

Design References

• Apple Human Interface Guidelines: Gestures
• Apple Human Interface Guidelines: Apple Pencil and Scribble
• Apple: Adopting hover support for Apple Pencil
• Apple: Coordinating multiple gesture recognizers
• Apple Human Interface Guidelines: Buttons
• Shapr3D: Keyboard shortcuts, gestures, and hotkeys
• Shapr3D: Navigation presets