3.3 Variable AutoEncoders & Gen Adversarial Network

Sat, 02 May 26

3.5 Gen AI: Models & Architecture Final

Introduction to Autoencoders

• Neural networks that learn to compress data into condensed representation and then reconstruct it to understand and capture essential features
• Unsupervised learning models used for dimensionality reduction, data compression and feature extraction
• Three core components:
  • Encoder: maps input data into lower-dimensional representation
  • Latent Space: data at most compressed form
  • Decoder: maps lower-dimensional representation back to original input data
• Reconstructs images to be as close to original image as possible
• Challenges with standard autoencoders:
  • Limited generative capabilities for data generation
  • Learn oversimplified representations during training
  • Struggle with inherent variability and randomness in data

Variational Autoencoders (VAEs)

• Autoencoders with enhanced generative capabilities that overcome standard autoencoder limitations
• Designed to learn probability distributions of input data through probabilistic compression instead of deterministic compression
• Generate new data samples similar to training data, making them powerful for generative tasks
• Create robust and generalizable representations for data with variability and complexity

VAE Architecture Components

• Encoder: compresses input data and produces parameters (mean and variance) of probability distribution
• Latent Space: compressed representation in probabilistic distribution form, encodes essential features for reconstruction
• Decoder: reconstructs input data from latent space representation
• Reconstruction Loss: measures decoder’s reconstruction quality using Mean Squared Error (MSE) or Cross-entropy loss
• KL Divergence: measures divergence of latent space distribution from prior normal distribution

VAE Training Process

1. Data Collection: gather large dataset representing target domain
2. Encoding: encoder maps input data (x) to latent space (z), learns mean and variance of Gaussian distribution
3. Sampling: model samples from learned distribution in latent space, introduces randomness for generative capabilities
4. Decoding: decoder generates new data samples, maps latent representation back to data space
5. Objective Function: optimize two components:
   • Minimize reconstruction error between input and generated data
   • Minimize KL divergence between learned distribution and standard Gaussian distribution
6. Training and Backpropagation: computes gradients for encoder and decoder parameters, updates parameters to minimize objective function

Generative Applications of VAEs

• Image Generation: create new realistic images, visual artworks, game assets and characters, medical images for research
• Anomaly Detection: identify outliers in datasets, spot unusual financial transactions, enhance security systems, improve manufacturing defect detection
• Drug Discovery: accelerate potential drug identification, design molecules with specific properties
• Data Imputation: fill missing or incomplete data, particularly valuable where missing data complicates decision-making

VAE Drawbacks

• Tend to produce blurry and unrealistic outputs compared to other generative models
• GANs known for producing higher-quality, sharp and realistic outputs, particularly in image generation

Generative Adversarial Networks (GANs)

• Deep learning architectures using convolutional neural networks for generative modeling
• Generate highly realistic samples with sharp details and intricate features
• Excel at capturing high-frequency details and generating more realistic, diverse samples than VAEs
• Produce images that capture complexity and variability of real data

GAN Architecture

• Uses two neural networks in adversarial relationship:
  • Generator: receives noise vector input, creates sample images to deceive discriminator
  • Discriminator: functions as binary classifier, provides probabilities from 0 to 1
    • Result closer to 0 indicates higher likelihood sample is fake
    • Result closer to 1 indicates higher likelihood sample is real
• Both networks implemented using CNNs for image-related tasks
• Forms zero-sum game where one network’s progress comes at expense of the other

GAN Training Process

1. Initialize both generator and discriminator networks with random weights
2. Train discriminator network on batch of real data samples and batch of generated samples
3. Train generator network to create new data samples that can deceive discriminator
4. Repeat steps 2 and 3 until networks achieve convergence

GAN Applications and Benefits

• StyleGAN: NVIDIA-developed network generating highly realistic, customizable synthetic human faces
  • Reference: https://user-images.githubusercontent.com/6625384/64915614-b82efd00-d730-11e9-92e4-f3a6de1a5575.png
• Industrial Use Cases:
  • Virtual clothing try-on: customers upload photos to see how clothing items look
  • Customized shopping: create personalized shopping experiences with tailored recommendations
• Operate in unsupervised learning framework without requiring labeled data
• Enable image-to-image translation tasks (satellite to maps, black-and-white to color)
• Support style transfer allowing synthesis of images in specific artistic styles
• Research reference: https://arxiv.org/abs/1906.00446?ref=assemblyai.com

GAN Drawbacks

• Difficult to train due to adversarial relationship between generator and discriminator
• Prone to mode collapse where model generates only limited subset of samples
• Requires careful optimization of both networks simultaneously

Technical Implementation Details

• Demonstrated VAE implementation using TensorFlow and Keras frameworks
• MNIST dataset used for training examples (took approximately 30 minutes for basic model)
• Training involves hyperparameter tuning: learning rate, batch size, epochs, noise dimensions
• Early stopping techniques prevent overfitting when loss stops improving
• Hyperparameter optimization automates finding best parameter combinations

---