Research Portfolio

Large multimodal models (LMMs) like GPT-4V can process text, images, and video — but unlike humans, their outputs often lack common sense and can be inconsistent across modalities. In multi-label video object recognition, a model might identify an object in one frame but miss it in the next, even when frames are nearly identical.

Automated metrics like F1 or average precision cannot capture such inconsistencies, making human evaluation essential. But there was no simple tool that let non-experts assess whether a model was actually performing reliably.

IKIWISI (I-Know-It-When-I-See-It) is a lightweight, intuitive tool that visualizes LMM prediction inconsistencies across video frames via binary heatmaps (Figure 1):

• The user selects objects of interest and a video from dropdowns.
• Up to 16 keyframes are displayed in an image container for inspection.
• A binary heatmap represents model predictions — each row is an object, each column a video frame.
• High-level patterns in the heatmap let users quickly judge model reliability without any ML expertise (Figures 12 and 13).

• In a user study with 15 participants, the heatmap visualization enabled identification of subtle inconsistencies — models flipping answers across turns or hallucinating details.
• Participants' ratings of model performance correlated strongly with actual F1 scores.
• Users inspected only a tiny fraction of frames to judge a model, thanks to high-level heatmap patterns.
• Even non-technical users could identify inconsistency patterns and contribute to model assessment workflows.

IKIWISI is among the first tools to enable intuitive inconsistency diagnosis across LMM outputs without requiring code or model access. It supports faster and more inclusive evaluation workflows for researchers, QA teams, and designers, and can be integrated into model audit pipelines and prompt refinement tools.

Blind users often face major challenges when navigating modern graphical user interfaces, especially when interface elements are deeply nested, unlabeled, or visually grouped without accessible structure.

Traditional screen readers and keyboard shortcuts are often linear and slow, making it difficult to work efficiently in complex software or perform tasks such as remote technical support. These barriers can increase reliance on sighted assistance and reduce digital autonomy.

Wheeler is a mouse-shaped device with three independently rotatable wheels, where each wheel controls a different axis or level of navigation (Figure 1).

• H-Nav Mode: supports hierarchical navigation through app structure, such as window → panel → element (Figure 2).
• 2D-Nav Mode: supports spatial navigation, where two wheels control horizontal and vertical cursor movement and the third wheel controls cursor speed (Figure 5).
• The device uses rotary encoders along with haptic and audio feedback.
• It was iteratively co-designed with blind users through participatory design sessions.

• Two controlled lab studies with 12 blind participants.
• A 6-month diary study with a blind co-author using Wheeler in real-world scenarios.

• Reduced navigation time by up to 40% in hierarchical tasks compared to screen reader + keyboard interaction.
• Helped users access unlabeled or hidden UI elements such as sliders, tooltips, and aria-hidden items.
• Showed real-world utility in productivity tasks, web browsing, and remote collaboration.
• Users reported improved agency, lower frustration, and quick learnability.

Wheeler introduces a new rotary, parallel input paradigm for non-visual interaction. It enables faster and more structured access to complex application hierarchies, supports interaction with otherwise inaccessible interface components, and offers a low-cost augmentative tool that works alongside screen readers.

Low-vision users often rely on screen magnifiers to interact with smartphone interfaces. But conventional magnifiers enlarge both useful content and redundant whitespace, which forces excessive panning and frequently causes loss of context.

Because these tools do not reason about layout structure, they can make everyday mobile interaction slower, more effortful, and more cognitively demanding.

SpaceXMag is an optimization-based system that automatically compacts whitespace in smartphone UIs while preserving structural relationships between interface elements.

• Operates on DOM-based UI metadata, not raw pixels.
• Uses nonlinear optimization with GEKKO/IPOPT to restructure layouts under element constraints.
• Compacts screens through a bottom-up hierarchical traversal for local flexibility and global consistency (Figure 17).
• Supports fullscreen, window, and fisheye magnifier modes (Figure 1).
• Includes a rendering backend to reconstruct the modified layouts visually.

• User study with 11 low-vision participants.
• Large-scale testing on 16,500+ screens from the RICO-SCA dataset.

• 28.13% faster in overview tasks than baseline magnifiers.
• 42.89% faster in target acquisition tasks.
• Users reported better screen awareness, lower zooming effort, and higher satisfaction, especially in window mode.
• Saved an average of 47.17% whitespace across 16.5K+ real app screens.
• Median runtime was 1.44 seconds per screen, up to 60× faster than prior solutions like SUPPLE.

SpaceXMag demonstrates that DOM-aware, optimization-driven layout restructuring can make magnification substantially more usable in real-world mobile interfaces. It points toward smarter adaptive accessibility tools that preserve context instead of simply enlarging pixels.

In classroom settings, general-purpose LLMs can easily over-help, provide direct solutions too quickly, or ignore the structure and expectations of a specific assignment. This makes it hard for instructors to use AI support without undermining students' own reasoning process.

Instructors also need a practical way to adapt AI behavior across assignments, manage student access, and review interactions in a structured classroom environment.

TeachPilot (shown in the demo above) is a teacher-guided LLM assistant that changes behavior per assignment and applies instructor-defined guardrails before sending requests to the model.

• Uses a two-part prompting structure: a base system prompt plus an assignment-specific prompt.
• Supports assignment-scoped conversations, so each chat is tied to the active coursework context.
• Allows text, image, and file uploads in chat when using compatible vision-language models.
• Provides admin controls for assignments, prompts, users, and conversation browsing.
• Provides a student-facing chat experience with copy-message and copy-conversation utilities.

• Built with Streamlit and Ollama Cloud.
• Persists chats and metadata in Supabase Postgres, with optional Supabase Storage for uploads.
• Supports role-based use: admin manages assignments/prompts/users, while students use chat only.
• Processes uploaded images and files before composing the final model request.

• Creates a more controlled classroom AI workflow centered on reasoning-first support instead of unrestricted answer delivery.
• Gives instructors direct control over assignment framing, help constraints, and conversation review.
• Makes it feasible to deploy course-specific AI support with persistent history, transcript access, and manageable administration.

TeachPilot shows how classroom AI can be structured as an instructor-shaped system rather than a generic chatbot. By combining assignment-aware prompting, role-based controls, and persistent conversation management, it supports more accountable and pedagogically aligned AI use in education.

Blind and low-vision (BLV) individuals face major challenges when navigating urban and indoor environments, but existing computer vision datasets often miss many objects that are critical for safe movement.

Common datasets such as ImageNet and MS-COCO do not adequately represent hazards and navigation-relevant objects like overhanging branches, sidewalk pits, or bus stops, limiting the usefulness of current vision systems for accessibility.

This project created an accessibility-centered object taxonomy and benchmark specifically for BLV navigation.

• Analyzed 21 publicly available navigation videos involving BLV individuals.
• Identified 80+ candidate objects that affected navigation through interaction, guidance, or hazard potential.
• Refined the list through a 6-person focus group with blind, low-vision, and sighted accessibility experts.
• Produced a final set of 90 objects across 15 semantic categories (Table 1, Figure 2).
• Labeled 31 video segments with object presence and absence annotations.
• Benchmarked seven vision models including detection, segmentation, and open-vocabulary systems (Figure 3).

• Modern vision models performed poorly on many accessibility-critical objects (Figure 3).
• Detection and segmentation models typically recognized only 12–15 objects (Figure 3).
• Open-vocabulary models performed better, but still had incomplete coverage.
• BLIP achieved the best reported result with an F1 score of 0.48, yet still missed important safety-relevant objects such as barrier stumps and tree branches (Figure 3).

This work highlights a major gap between state-of-the-art vision systems and the real object awareness needed for safe BLV navigation. By releasing a benchmark dataset and object taxonomy grounded in accessibility needs, it lays the foundation for more inclusive navigation aids and future accessibility-focused vision models.

Blind screen reader users rely on keyboard navigation to interact with desktop applications, but navigating menus, toolbars, dialogs, and nested interface structures is often slow, inconsistent, and frustrating.

Existing accessibility evaluation methods usually focus on compliance checks or labor-intensive user studies. They do not provide a scalable way to estimate perceived accessibility — how accessible an application actually feels to blind users during real non-visual interaction.

This project introduces a probabilistic interaction framework for estimating perceived accessibility from the structure of desktop interfaces and keystroke-based navigation behavior.

• Built from findings of a formative study with 11 blind participants (Figure 2).
• Uses UI hierarchies extracted through accessibility APIs, without needing pixel data or screen recordings.
• Models navigation effort through a probabilistic interaction model and keystroke cost analysis.
• Defines three task-agnostic metrics: Complexity, Coverage, and Reachability.
• Benchmarks accessibility across 11 commonly used Windows applications (Figure 10).

• The metrics revealed clear perceived-accessibility differences across applications.
• Notepad showed low complexity (1.63), while Microsoft Word showed substantially higher complexity (3.51).
• Participants described the metrics as intuitive and reflective of their actual navigation effort.
• Applications with high complexity and low coverage were consistently experienced as more frustrating and difficult to use.

This work provides an automated, user-centered way to estimate perceived accessibility for non-visual desktop interaction. It helps developers and auditors compare applications, identify concrete navigation bottlenecks, and move accessibility evaluation toward scalable, interface-aware assessment rather than compliance alone.

Blind and low-vision users face major barriers in Unity-based 3D and VR applications because these apps often lack screen reader support, accessible UI components, and effective keyboard navigation.

Existing accessibility plugins are fragmented, visually oriented, and usually must be built into the game during development, which makes them hard to retrofit and inadequate for non-visual interaction in existing immersive applications.

OpenAccAPI is an extensible accessibility API for 3D and VR applications that exposes UI and scene elements to assistive technologies and supports retrofitting accessibility into existing Unity apps.

• Provides access to scene and interface elements for assistive tools.
• Enables features such as text-to-speech, UI text highlighting, non-visual navigation, and cursor orientation feedback (Figure 6).
• Includes a reference implementation for Unity.
• Supports plugin injection into existing Unity games in three clicks (Figure 7).

• Tested on 15 popular Unity games on Steam (Table 2).
• Most features worked smoothly in 12 of the 15 games (Table 2).
• Small-scale studies with blindfolded users suggested better spatial awareness and navigation capability in 3D and VR settings.

OpenAccAPI introduces a comprehensive accessibility API for 3D and VR apps aimed at non-visual interaction. It creates a practical path toward more accessible immersive software and lays groundwork for future research in accessible XR design and agent-based non-visual navigation.

Video investigations are often slow, repetitive, and difficult to standardize. Analysts need to search long recordings, identify meaningful incidents, compare events across cameras, and document evidence in a repeatable way.

Many practical workflows also need to remain lightweight and explainable, without requiring expensive model training or opaque end-to-end systems.

ForenSight (shwon in the demo above) is a local-first, no-training investigation pipeline that combines pretrained vision embeddings with lightweight, explainable rules for search, review, and incident analysis.

• Text-to-timestamp retrieval: CLIP text embeddings are matched against FAISS indexes built over sampled video frames.
• Event timeline: detects change-point spikes from embedding deltas to surface likely moments of interest.
• ROI and zone rules: supports motion detection, optional person enter/exit, and loitering within user-defined regions.
• Tripwire rules: detects line-crossing events with optional direction constraints.
• Policy tagging: converts video-relative timestamps to real datetimes and can automatically label events such as after-hours activity.
• Multi-camera correlation: links incidents across camera feeds using time-window gating and CLIP similarity of evidence frames.

ForenSight demonstrates a practical middle ground between manual video review and heavyweight learned surveillance systems. It emphasizes grounded retrieval, explainable event logic, reproducible case documentation, and local-first deployment for more transparent investigation workflows.

Large language models can retain undesirable behaviors, biased patterns, or targeted knowledge that developers may later want to remove. Full retraining is expensive, slow, and often impractical for iterative experimentation.

This project builds a parameter-efficient unlearning workflow using synthetic data generation and LoRA-based fine-tuning.

• Generates targeted synthetic training data to represent the behavior or knowledge to be removed.
• Uses PEFT / LoRA adapters instead of full-model retraining.
• Supports multi-GPU fine-tuning for efficient experimentation at larger model scales.
• Separates data generation, adapter training, and post-unlearning evaluation into a modular pipeline.

This project explores a practical path toward controllable, low-cost machine unlearning for large language models. By combining synthetic data with parameter-efficient fine-tuning, it supports faster experimentation on selective forgetting, safety adaptation, and post-hoc model editing.

Current video-language models are built around rendered pixels or estimated features such as flow and depth. In complex 3D and virtual environments, that makes scene understanding expensive and often brittle, because the models do not directly observe the object-level dynamics that actually drive the scene.

Interactive worlds already contain rich internal state — object identities, positions, scales, visibilities, hierarchies, colliders, and activation flags — but that information is rarely exposed in a form that language models can reason over cleanly.

OTN converts raw engine telemetry into a structured, language-ready representation of scene dynamics. Instead of asking a model to infer motion from pixels, it provides an object-centric motion grammar that explicitly summarizes how entities appear, move, and interact over time.

• Logs per-object telemetry such as positions, scales, visibilities, hierarchy, and semantic properties.
• Builds both quantitative and symbolic layers, including motion statistics, phase timelines, interaction summaries, outlier events, and deterministic textual narratives.
• Represents each salient object through a compact narrative table that a language model can consume directly.
• Uses a Unity plugin for logging, while keeping the overall representation general enough for other engines with comparable state access.

• A relatively small language model, around 7B parameters, can use OTN text alone for zero-shot scene description and temporal reasoning.
• The paper reports performance that is comparable to, or better than, much larger video-language systems while using far less compute and latency.
• Light fine-tuning on structured OTN data further improves temporal coherence, reference consistency, and object grounding.

OTN offers a different interface between 3D simulation and language understanding: instead of scaling bigger pixel-based models, it leverages engine-native structure to make reasoning cheaper, more explicit, and easier to audit.

Modern video object segmentation models use memory modules and propagation mechanisms to maintain temporal correspondence, but they are still trained mostly with frame-level losses such as cross-entropy and Tversky loss.

That creates a mismatch: the architecture is temporal, but the training signal is largely static. As a result, models often fail in scenarios where temporal consistency matters most, especially with small objects, heavy occlusion, and objects that disappear and later re-emerge.

This project introduces an explicit temporal consistency loss that can wrap around existing semi-supervised VOS training pipelines without requiring architectural changes (Figure 2).

• Represents per-frame object presence and absence as binary vectors over all tracked objects.
• Models temporal consistency through a Hidden Markov Model.
• Uses a differentiable soft Viterbi-style formulation to estimate the most likely temporally consistent sequence.
• Combines two sub-losses: one based on Hamming-distance deviation and one based on a probability ratio between the ground-truth and inferred temporal sequence.
• Integrates with standard losses through a weighted combination instead of replacing pixel-level supervision.

• Improved overall J & F by roughly 0.9–1.3% on AOT-T across DAVIS, YouTube-VOS, and LVOS.
• Improved DeAOT-T by 1.9% on DAVIS while remaining competitive on YouTube-VOS.
• Produced larger gains of roughly 3–4% on the hardest cases: small objects, occlusion, and disappearance/reappearance (Figure 1).
• An ensemble of baseline and temporal-loss-trained models achieved up to 3.4% improvement on DAVIS and additional gains on YouTube-VOS and LVOS.

This work shifts part of temporal reasoning from architecture design into the training objective itself. By making temporal consistency an explicit optimization target, it offers a simple and reusable way to improve VOS robustness in real-world challenging videos without redesigning the underlying model.

PAALSS-style analysis of aided AAC transcripts is detail-heavy and formatting-sensitive. Researchers and clinicians often need to clean transcripts, structure utterances consistently, prompt the model carefully, and turn the output into a usable draft report.

PAALSS Analyzer (shown in the demo above) is a lightweight Streamlit app that turns transcript analysis into a controlled workflow rather than an ad hoc chat interaction.

• Uploads .docx or .txt transcripts.
• Extracts and pre-fills a numbered transcript block of Enunciados.
• Lets the user edit the transcript before analysis so the model analyzes exactly what is shown in the editor.
• Lets the user edit a base system prompt to control report format and level of detail.
• Sends transcript plus prompt to a selected Ollama model and returns a PAALSS-style report.

PAALSS Analyzer packages transcript parsing, prompt control, bilingual interface support, model selection, and document export into a focused research workflow. It is best framed as a drafting-support tool for structured AAC analysis, not as a diagnostic system.