Debabrata Mishra

Debabrata Mishra

ML Research Engineer

Blog posts

2025

Mixture of Experts

Published:

As Large Language Models (LLMs) continue to grow in size and complexity, a fundamental challenge emerges: how can we scale model capacity while maintaining computational efficiency? The traditional approach of simply adding more parameters to dense networks quickly becomes prohibitively expensive, both in terms of computational cost and memory requirements. This is where Mixture of Experts (MoE) architectures shine, offering an elegant solution that dramatically increases model capacity while keeping computational costs manageable.

Key-Value caching

Published:

What is Key-Value caching?

Key-value caching, as an optimization technique, focuses on improving the efficiency of the inference process in Large Language Models(LLMs) by reusing previously computed states. In simple terms, it’s a way for the model to “remember” previous calculations to avoid re-computing them for every new word it generates.

KL Divergence

Published:

Understanding Kullback-Leibler divergence

In today’s rapidly evolving artificial intelligence landscape, one mathematical concept stands as a cornerstone across countless applications: the Kullback-Leibler (KL) divergence. With the rise of large language models (Thanks to ChatGPT!), we’ve been fastforwarded into the future, thanks to the countess innovations that has gone in the past. From training large language models to detecting anomalies in real-time data streams, KL divergence has become the silent engine powering some of the most sophisticated AI systems. As we witness unprecedented advances in machine learning, understanding this fundamental measure of distributional difference has never been more crucial for data scientists and ML researchers.

Reinforcement Learning: Present and Future

Published:

Hello World! Welcome to my first blog post. Today, I’m diving into a topic that’s shaping the future of AI — Reinforcement Learning (RL). Whether you’re an AI enthusiast, a curious reader, or just someone wondering how robots and algorithms are getting so smart, this post is for you.