CS671 · Deep Learning and Applications

EEG-Based Visual Brain Decoding

Temporally coherent video generation from neural signals using
Transformer Encoders, Multi-Path Conditioning, and Stable Diffusion.

Group: 22

Professor: Aditya Nigam

Mentor: Jyoti Nigam

02 / The Team

Group 22 Members

Vipresh Gupta

B24412

Vishal Meena

B24413

Sarthak Kardam

B24404

Manan Sahni

B24397

Raghav Bansal

B24402

Shivam Gami

B24405

Shivam Singla

B24406

Kamesh Singh

B23317

Milan Jadav

B23134

Rishab Bagul

B24151

03 / Problem & Motivation

Decoding the Visual Cinema

Evolving Scenes

Our visual experiences are composed of continuously evolving scenes caused by the movement of objects and viewing perspective. To investigate the mechanism of our visual system, neuroimaging techniques (fMRI, MEG, EEG) are used to analyze brain activities.

The Temporal Resolution Gap

Recently, works have reconstructed high-quality two-second videos from single fMRI frames. However, limited by the low temporal resolution of fMRI, these lack the ability of capturing high dynamic changes. Thus, we propose video reconstruction from EEG signals.

The Semantic & Temporal Gap

Mapping low-dimensional neural signals to the high-dimensional pixel space of video frames requires overcoming the semantic gap between neural activity patterns and perceptual representations.

Noisy Signals: EEG is notoriously noisy, non-stationary, and highly variable across subjects.
Temporal Consistency: Generating video demands architectures that explicitly model cross-frame dependencies, absent from standard image models.

04 / Executive Summary

Abstract

Decoding human visual perception directly from brain activity stands as one of the most ambitious frontiers in computational neuroscience. This project, EEG2Video, proposes a novel multimodal Deep Learning pipeline for reconstructing dynamic visual stimuli (video) from non-invasive EEG signals.

Leveraging the SEED-DV dataset, raw 64-channel EEG recordings are transformed via STFT into frequency spectrograms and encoded through a Transformer Encoder. A novel Dynamic-Aware Noise Adding (DANA) module adaptively conditions latent vectors based on EEG-derived temporal motion labels. Video generation is performed using a fine-tuned Stable Diffusion v1.5 architecture augmented with a TemporalUNet backbone to ensure inter-frame coherence.

05 / Background

Why EEG over fMRI?

High Temporal Resolution

fMRI has excellent spatial resolution but poor temporal resolution (captures changes every ~2 seconds). EEG offers millisecond-level temporal resolution, making it uniquely suited for capturing the fast neural dynamics required for video reconstruction.

Accessibility & Portability

fMRI and MEG require massive, stationary, and expensive medical equipment. EEG is low-cost, portable, and non-invasive, paving the way for practical, real-world BCI applications outside clinical settings.

EEG Setup

64 Channels 200Hz Sampling Non-Invasive BCI

06 / Literature

State-of-the-Art Baseline

EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals

Authors: Xuan-Hao Liu et al.

Conference: NeurIPS 2024

Dataset Used: SEED-DV (20 subjects, 1400 EEG-video pairs)

Architecture in Brief

A Seq2Seq Transformer that densely maps temporally-windowed EEG embeddings (extracted via a GLMNet) to video latents. It utilizes a Dynamic-Aware Noise-Adding (DANA) module and a fine-tuned inflated Stable Diffusion (Tune-A-Video) model guided by CLIP-aligned embeddings.

        
        Key Contributions
        Introduced the first large EEG-video paired dataset (SEED-DV) with rich meta-annotation.
Proposed GLMNet encoder combining global and local (occipital) EEG features.
First to leverage EEG's high temporal resolution via Seq2Seq alignment for dynamic video reconstruction.

07 / Literature

EEG2Video Results & Limitations

Reported Metrics (Full 40-class)

Metric	EEG2Video (SOTA)
2-way video semantic acc.	79.8%
40-way video semantic acc.	15.9%
SSIM (pixel-level)	0.256

Note: On 10-class subset, SSIM reaches 0.300.

Identified Limitations

Single Session: Each subject recorded in only one session, lacking stable temporal neural patterns.
Subject-Dependent Only: Cross-subject generalization is entirely unexplored (a major gap for EEG).
Preliminary Quality: Overall reconstruction performance is preliminary (SSIM 0.256).

08 / Background

Related Works & Foundations

Neural Decoding from EEG

Early approaches were limited to binary classification (e.g., motor imagery). Recent advances enabled emotion recognition and coarse scene reconstruction, but full video decoding remained unexplored due to temporal complexity.

Latent Diffusion Models (LDMs)

Rombach et al. (2022) proposed LDMs as a computationally efficient alternative to pixel-space diffusion by operating in the compressed latent space of a pre-trained VAE, serving as our powerful generative base.

Tune-A-Video

Wu et al. (2023) demonstrated that image diffusion models can be adapted for video by injecting lightweight 1D temporal attention layers while freezing spatial weights, preserving rich visual priors without catastrophic forgetting.

CLIP and ViT Representations

Radford et al. and Dosovitskiy et al. proved that CLIP text encoders and Vision Transformers produce semantically rich embeddings. We leverage both modalities as complementary conditioning streams.

09 / Our Contribution

How We Differentiate

Dimension	SOTA (EEG2Video)	Other Work (Tune-A-Video/LDM)	Our Implementation
Temporal Modeling	GLMNet + Seq2Seq	1D Temporal Attention (Text to Video)	TemporalUNet (Interleaved 1D Attn) driven by EEG
Conditioning	CLIP primarily	Text prompts only	Tri-Path: Semantic (CLIP) + Structural (ViT) + DANA
Noise Initialization	Standard Gaussian	Standard Gaussian	DANA (Motion-conditioned variance based on dynamics)
Preprocessing	Global-Local Mixture	N/A	STFT Spectrograms + Transformer Encoder

10 / Data

Dataset & Preprocessing

64

Channels

200Hz

Sampling

500

Timesteps

          
          Preprocessing Steps
          1. Bandpass Filtering: Zero-phase Butterworth (0.5–50 Hz) removes drift and high-frequency noise outside physiological EEG bands.
2. Artifact Removal: Independent Component Analysis (ICA) suppresses ocular (EOG) and muscular (EMG) artifacts.
3. Epoch Segmentation: Continuous recordings segmented into 500 time-step epochs time-locked to video stimuli.
4. Standardization: Per-channel z-score normalization ensures zero mean and unit variance.

        

SEED-DV Ground Truth Video

11 / Architecture

Full Pipeline Overview

A 5-stage hybrid deep learning architecture designed specifically for mapping noisy brain signals to temporally coherent video latents.

          1. STFT Preprocessing & Tokenization
Transformer Encoder Base
Tri-Path Extraction (CLIP + ViT + Classifier)
DANA Latent Stabilization
TemporalUNet Generation & VAE Decode

        

Interactive Pipeline Architecture

Explore the high-resolution architecture diagram on our main website interface.

View Interactive Pipeline →

12 / Methodology

Stage 1: Preprocessing (STFT)

Time-Frequency Representation

Raw 64-channel EEG signals are inherently noisy and complex in the time domain. We convert overlapping windows into Short-Time Fourier Transform (STFT) spectrograms.

This extracts power spectral density across standard frequency bands (Delta, Theta, Alpha, Beta, Gamma) into a structured 64 × 64 spatial representation.

$$ \text{STFT}_i[m, k] = \sum_{n=0}^{N_w - 1} x_i[n + mH]\, w[n]\, e^{-j2\pi kn / N_w} $$

13 / Methodology

Stage 2: Transformer Encoder

T

EEGNet-Inspired Backbone

4 Layers · 8 Attention Heads

The STFT maps are patchified and passed through a Vision Transformer style encoder. This captures global spatial-spectral relationships across different brain regions.

          Output Tensor: 8 × 8 × 512 embedding

          Acts as the foundation for the three downstream conditioning paths.

14 / Methodology

Stage 3a: Semantic Path (CLIP)

Semantic Alignment

A TextMLP maps the base EEG embedding into a CLIP-compatible text space. This ensures the generated video matches the high-level concept of what the subject was viewing.

Fix for Mode Collapse: We resolved early mode collapse by injecting raw GT text into the CLIP training alignment.

Cosine Alignment & Thresholding

We use Cosine Similarity loss during training to align the predicted embedding $z_e$ with the ground-truth text embedding $z_t$.

$$ \mathcal{L}_{sim} = 1 - \frac{z_e \cdot z_t}{\|z_e\| \|z_t\|} $$

Threshold: $S_c \ge 0.7$

Samples with alignment meeting the threshold provide strong semantic guidance.

15 / Methodology

Stage 3b: Structural Path (ViT)

Spatial & Compositional Guidance

While CLIP provides semantic concepts ("a dog running"), the ViT path preserves structural layout and composition from the visual cortex signals.

Produces visual latents of shape 6 × 4 × 32 × 32 to guide the structural formation of the generated frames.

ViT Latents

ViT Output Attention

16 / Methodology

Stage 3c: Dynamic Classifier + DANA

Temporal Dynamics (Fast/Slow)

A binary classifier predicts the motion profile of the scene optimized via Cross-Entropy.

$$ \mathcal{L}_{CE} = - \frac{1}{N} \sum_{i=1}^{N} y_i \log(\hat{y}_i) + (1-y_i) \log(1-\hat{y}_i) $$

Validation Accuracy: 58.27%

DANA Latent Stabilization

Brain signals are noisy. The DANA module adaptively modulates the VAE latents based on the dynamic classifier's output, preventing mode collapse.

$$ z_{\text{DANA}} = z + \sigma(d)\cdot\epsilon $$

17 / Methodology

Stage 4: TemporalUNet Generation

A

Frozen Spatial Backbone

Preserves SD v1.5 generative image priors

B

Trainable Temporal Attention

Interleaved 1D layers for inter-frame coherence

$$ \text{TemporalAttn}(\mathbf{h}) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V $$

Diffusion Training Objective

$$ \mathcal{L}_{\text{MSE}} = \mathbb{E}_{z,\,\epsilon,\,t}\! \left[\left\|\epsilon - \epsilon_\theta\!\left(z_t, t, c_{\text{text}}, c_{\text{vis}}, d\right)\right\|_2^2\right] $$

Diffusion MSE ≈ 0.0005

18 / Methodology

Inference Algorithm

Algorithm 1: EEG2Video Inference

Require: EEG epoch $x_{\text{EEG}} \in \mathbb{R}^{64 \times 500}$, DDIM steps $T=20$

1.

Compute STFT: $S \leftarrow \text{STFT}(x_{\text{EEG}}) \in \mathbb{R}^{64\times64}$

2.

Encode: $h \leftarrow \text{TransformerEncoder}(S) \in \mathbb{R}^{8\times8\times512}$

3.

Extract: $c_{\text{text}} \leftarrow \text{CLIPEncoder}(h)$ , $c_{\text{vis}} \leftarrow \text{ViTEncoder}(h)$

4.

Classify: $d \leftarrow f_\phi(\text{Pool}(h))$

5.

Sample Noise: $z_T \sim \mathcal{N}(0, \mathbf{I})$ of shape $(B, 6, 4, 16, 16)$

For $t = T, T-1, \ldots, 1$ do

6.

$\hat\epsilon \leftarrow \epsilon_\theta(z_t, t, c_{\text{text}}, c_{\text{vis}}, d)$ // TemporalUNet step

7.

$z_{t-1} \leftarrow \text{DDIM\_Step}(z_t, \hat\epsilon, t)$

End For

8.

Decode: $\hat{X} \leftarrow \{\mathcal{D}(z_0^{(f)})\}_{f=1}^{6}$

Return: 3-frame video clip $\hat{X}$ at 3 FPS

19 / Results

Training Loss Convergence

TemporalUNet Phase 4 Validation

The model closely approaches the architectural target of ≈0.0005 MSE over 5 epochs, validating tensor shape contracts and gradient flow.

20 / Results

Quantitative Metrics

0.6967

SSIM ↑

29.17 dB

PSNR ↑

0.3526

LPIPS ↓

24.50

FID ↓

Ablation Study

Configuration	SSIM	PSNR	LPIPS
Full Model (DANA + ViT)	0.6967	29.17	0.3526
w/o DANA	0.5459	20.14	0.4004
w/o Vision Transformer	0.4647	16.53	0.5400

21 / Results

Qualitative: GT vs Generated

GTGen

22 / Applications

Real-World Impact & Applications

Medical Rehabilitation

Brain-Computer Interfaces (BCIs) that can reconstruct perceptual states hold immense potential to revolutionize assistive communication for locked-in patients and individuals with severe motor disabilities.

Next-Gen Interaction

Robust visual decoding paves the way for direct thought-to-screen media generation and entirely new forms of seamless human-computer interaction in mixed reality environments.

Cognitive Neuroscience

Decoding the fast temporal dynamics of visual scenes deepens our fundamental understanding of the neural correlates of visual consciousness and how the brain processes continuous motion.

23 / Discussion

Limitations & Future Work

Current Limitations

Synthetic Integration: Phase 4 architecture was validated using synthetic tensors; full end-to-end integration with real live EEG embeddings awaits Phase 5.
Resolution Limits: Outputs are currently constrained to 128×128 px. Scaling to high-definition requires significantly more VRAM and compute.
Frame Capacity: Current architecture generates 3-frame sequences; longer horizons need hierarchical modeling.
Subject Variability: EEG signals vary wildly between subjects; personalized adaptation is currently missing.

        
        Future Directions
        Phase 5 Integration: Full SEED-DV benchmarking combining visual and semantic paths from Sub-teams 3 & 4.
Classifier-Free Guidance (CFG): Implementing CFG natively with EEG conditioning to boost generation fidelity.
Resolution Scaling: Integrating cascaded diffusion or super-resolution latent upscalers.
Subject Personalization: Implementing Few-Shot LoRA fine-tuning for specific subjects to combat variability.

      

Thank You

Questions & Discussion

Key References

Liu, S., et al. (2024). EEG2Video: Towards Decoding Dynamic Visual Perception from EEG. NeurIPS.
Wu, J.Z., et al. (2023). Tune-A-Video: One-Shot Tuning of Image Diffusion Models. ICCV.
Rombach et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR.

Song, J., et al. (2021). Denoising Diffusion Implicit Models (DDIM). ICLR.
Radford, A., et al. (2021). Learning Transferable Visual Models From Natural Language. ICML.
Dosovitskiy, A., et al. (2021). An Image is Worth 16x16 Words: Vision Transformers. ICLR.

← Back to Website