large language models - Langformers Blog

Langformers Blog

Sign in Subscribe

large language models

A collection of 8 posts

BPE Tokenizer: Training and Tokenization Explained

large language models

BPE Tokenizer: Training and Tokenization Explained

In my previous blog post, I explained in detail how large language models (LLMs) work, using analogies and maths. I received many requests for a simpler explanation of how a tokenizer is trained for such models, and how tokenization works. Hence, this blog post. How LLMs Work: A Beginner’s

How LLMs Work: A Beginner's Guide to Decoder-Only Transformers

large language models

How LLMs Work: A Beginner's Guide to Decoder-Only Transformers

A language model like GPT (which stands for Generative Pretrained Transformer) takes text, breaks it into tokens (words or subwords), converts those tokens into numbers, processes those numbers through layers of Transformer decoders, and finally outputs a probability distribution over all possible tokens in its vocabulary. It then selects the

large language models

What is RAG? A Beginner's Guide to Retrieval-Augmented Generation

Not long ago, all large language models (LLMs) had what we called a knowledge cutoff. This meant they only knew information up until a certain date — anything that happened after that, they simply couldn’t help you. Today, that’s changed, at least for cloud-based LLMs like ChatGPT. These models

Ollama and Local LLMs: Step-by-step Guide

large language models

Ollama and Local LLMs: Step-by-step Guide

Everyone loves ChatGPT, right? And hey, shoutout to DeepSeek Hero too. But what about the unsung heroes of the AI world—those open-source large language models (LLMs) quietly powering innovation behind the scenes? While models like Meta’s LLaMA might ring a bell for some, there’s a whole family

Fixed-size, Semantic and Recursive Chunking Strategies for LLMs

large language models

Fixed-size, Semantic and Recursive Chunking Strategies for LLMs

In modern Retrieval-Augmented Generation (RAG) systems, handling large documents efficiently is critical. Embedding models have token limits that, when exceeded, can lead to incomplete processing or errors. This is where chunking becomes critical. By breaking large documents into smaller, manageable pieces, chunking ensures that information remains accessible, relevant, and optimized

Data Labelling Using LLMs with Langformers

large language models

Data Labelling Using LLMs with Langformers

When most people think of Large Language Models (LLMs), they think of conversations, content generation, or summarization. But LLMs are also incredibly effective at data labelling — and now, with Langformers, you can easily utilize that power for you text labelling tasks. Whether you're preparing training data, building a

Build Your Own LLM-Powered Apps with Langformers: LLM Inference API

large language models

Build Your Own LLM-Powered Apps with Langformers: LLM Inference API

Today, we're excited to showcase another powerful feature of Langformers — LLM Inference via a REST API. Whether you’re building a chatbot, automating workflows, or adding AI magic to your product, Langformers gives you everything you need to bring large language models (LLMs) into your application stack effortlessly.

Chat with Open-Source LLMs Locally (Llama, Deepseek, Mistral)

large language models

Chat with Open-Source LLMs Locally (Llama, Deepseek, Mistral)

Large Language Models (LLMs) are everywhere — and everyone’s using them. But there’s a big issue emerging: people are feeding sensitive emails and company data into cloud-based LLM services like ChatGPT, DeepSeek, and others. Just to draft a few emails or generate some paragraphs, confidential information is getting handed