RNN Architecture Components:

• Hidden State: $h_t$ - Memory that carries information from previous timesteps
• Input Weight: $W_{input}$ - How much the current input affects the hidden state
• Hidden Weight: $W_{hidden}$ - How much the previous hidden state influences the current one
• Output Weight: $W_{output}$ - Projects hidden state to prediction space
• Biases: Constant terms added to the computation
• Activation Function: Non-linear transformation (tanh, ReLU, sigmoid)

RNN Formulas:
$h_t=f(W_{input}\cdot x_t+W_{hidden}\cdot h_{t-1}+b_{hidden})$
$y_t=W_{output}\cdot h_t+b_{output}$

Where $f$ is the activation function, $x_t$ is the input at time $t$, $h_{t-1}$ is the previous hidden state, and $y_t$ is the output prediction.

Note: The output layer separates internal memory ($h_t$) from task-specific predictions ($y_t$).

How to Use:

• Select a sequence to see different patterns (Fibonacci, linear, etc.)
• Adjust weights to see how they affect hidden state and output evolution
• Try different activation functions to understand their impact
• Use "Step Forward" to manually process each timestep
• Click "Train (BPTT)" to train using gradient-based backpropagation through time (the real algorithm)
• Click "Random Search" for comparison baseline (tries random weights without gradients)
• Toggle "Raw/Normalized Inputs" to scale sequences to [-1, 1] range

Sequence Types:
• Fibonacci: Each number is sum of previous two (requires richer state)
• Linear: Simple counting sequence
• Alternating: Pattern that switches between values
• Exponential: Powers of 2 sequence
• Sine Pattern: Smooth oscillating values

Understanding RNN Behavior:

• Input Weight ($W_{input}$): Higher values make current input more influential
• Hidden Weight ($W_{hidden}$): Controls memory retention. |$W_{hidden}$| > 1 can cause explosion, < 1 causes decay
• Output Layer: Separates internal memory ($h_t$) from predictions ($y_t$). Essential for flexible modeling
• Activation Functions:
- Tanh: Outputs between -1 and 1, standard for RNN hidden states
- ReLU: Only positive outputs, can cause exploding gradients in recurrence
- Sigmoid: Outputs between 0 and 1, prone to vanishing gradients
- Linear: No saturation but unstable without careful weight tuning

Training Insights:
• BPTT vs Random Search: "Train (BPTT)" uses gradient descent (the real algorithm used in practice). "Random Search" tries random weight combinations as a baseline for comparison. BPTT is vastly more efficient and shows why gradient-based optimization revolutionized neural networks.
• BPTT Algorithm: Backpropagation through time unfolds the network and computes gradients across all timesteps using the chain rule. Updates weights via gradient descent: w = w - lr * ∇w.
• Gradient Clipping: The demo clips gradients to [-5, 5] to prevent explosion, a critical technique for RNN training
• Learning Rate: Controls step size during gradient descent. Too high causes instability, too low causes slow convergence
• Normalization: Helps prevent activation saturation with bounded functions

Limitations:
• This demo uses a single scalar hidden unit. Real RNNs use vector hidden states for richer memory
• Fibonacci requires memory of two previous values - a single scalar $h_t$ is insufficient