Ways to join
The Backend Engineering Show with Hussein Nasser
$4.99/month
- Ad-free listening
For members only
548 episodes
- Developers will be prompters (and that is bad)I discuss the state of what will be happening in the coming few years and why I encourage the 3 of you who want to be elite engineers to keep their critical thinking sharp. To pause and think before prompting.
- AI will not replace elite engineersThis is a conversation with Rashed Hasan, a PhD Candidate from the University of Nebraska. We chatted about databases, ORMs and how to use AI without losing yourself. Rashed pagehttps://www.mdrashedulhasan.me/homepage
- Engineers will be out of the 'Loops'I discuss loops engineering and how it will numb creative thinking and critical system design for engineers.
- SQL was a mistakeI like SQL but the standard should have included ways to hint instead of the abstraction of "let me do it for you"
- Caching is a cop-outIn this episode I explore why you should take some time to examine the operation you are caching before doing so. And why caching is a cop-out
- How open ai agents "hacked" hugging faceHow open ai agents "hacked" hugging face
- How a query optimization gave birth to infinite scrollThe infinite scroll in social media as we know it today is despised by all. But it always start as a bad idea. It originated as a result of UI change to relief the backend database from inefficient queries. I explore this in this episode of the backend engineering. 0:00 Intro1:00 Classic Paged results 6:23 Limitations of paging9:00 The scroll interface 12:00 How TikTok to infinite scroll 14:00 The current state
- Postgres is half as fast in Linux 7.0An aws engineer discovered a 50% regression in postgres throughput while testing the new Linux 7.0 kernel. The cause turns out to be massive TLB and page faults exacerbated by Postgres process-based design. In this backend engineering show episode I dive deep into how this was discovered, the root cause and the possible fixes and workarounds. Intermediate and Advanced Backend Engineering Course Bundlehttps://courses.husseinnasser.com/bundleMy Book, Root Cause: Stories and Lessons from Two Decade
- Don't let AI rob youA discussion about why many engineers still love the struggle, the mistakes, and the process of figuring things out themselves. This is how we grow and get better and stronger. Letting AI do everything (even though it can’t) robs us this feeling..
- New Book: Root causeI wrote a new book that has been in the works for years. It is called Root Cause, and it is for those who enjoy the art of backend engineering.Early in my career, 20 years ago, I built backend and database applications without fully grasping their inner mechanics. Performance issues, race conditions, bugs, and even data corruption often left me lost.Since that day, I resolved to truly understand how systems work. From networking protocols and intermediary proxies to backend services and various
- 5 Backend Design Patterns for Managing Threads and SocketsIn this video I introduce 5 different design patterns for building backend applications. Each mode explains how a socket listener is established, a connections are established and how threads and connections are managed to read, write and process requests.
- Page TablesPage tables provide the mapping between virtual memory and physical memory for each process. This means it needs to be as efficient and as fast as possible. I explore the inner workings of page tables in this episode.0:00 Intro2:00 Virtual Memory ⁃ ⁃ 8:00 MMU10:00 Page Tables ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ 11:30 Single Table Byte Addressability ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ ⁃ 16:00 Single Table Page addressability ⁃ ⁃ ⁃ ⁃ ⁃ 19:00 Multi-level Paging (Radix tree) ⁃ ⁃ 31:00 Huge Tables ⁃ ⁃ 33:00 TLB ⁃ ⁃ Sum
- CPU and Kernel Page FaultsPage faults occurs when the process tries to access a memory that isn’t backed by a physical page kernel raises a fault which loads a page. It happens on first access, stack expansion, COW, swap and much more. However it comes with a cost. In this episode of the backend engineering show I dissect the need and the cost page faults in the kernel. 0:00 Intro 4:00 Virtual memoryAbstraction of physical memoryMemory sharingAllow more processes to run , unused go to diskNuma, kernel can place memory ne
- Amazon US-EAST-1 Outage in DetailsOn October 19 2025 AWS experienced an outage that lasted over a day, 10 days later we finally got the root cause analysis and we know exactly what caused the DNS to fail0:00 Summary 5:30 How did Dynamo lost its DNS?13:41 EC2 Errors 16:16 Network Load Balancer ErrorsRCA here https://aws.amazon.com/message/101925/
- Graceful shutdown in HTTPThere are cases where the backend may need to close the connection to prevent unexpected situations, prevent bad actors or simply just free up resources. Closing a connection gracefully allows clients and backends to clean up and finish any pending requests. In this episode of the backend engineering show I discuss graceful connections in both HTTP/1.1 via the connection header and HTTP/2 via the GOAWAY frame. 0:00 Intro4:58 Why shutdown connection? 6:46 HTTP/1.1 Graceful shutdown12:26 Cost of H
- Postgres 18 gets Async IOPostgres 18 has been released with many exciting features such as UUIDv7, Over explain module, composite index skip scans, and the most anticipated asynchronous IO with worker and io_uring mode which I uncover in this show. Hope you enjoy it0:00 Intro1:30 Synchronous vs Asynchronous calls3:00 Synchronous IO6:30 Asynchronous IO10:00 Postgres 17 synchronous io 17:20 The challenge of Async IO in Postgres 1820:00 io_method worker23:00 io_method io_uring29:30 io_method sync 31:08 Async IO isn’t done!
- Kernel level TLSFundamentals of Operating Systems Course https://oscourse.winktls is brilliant.TLS encryption/decryption often happens in userland. While TCP lives in the kernel. With ktls, userland can hand the keys to the kernel and the kernel does crypto. When calling write, the kernel encrypts the packet and send it to the NIC.When calling read, the kernel decrypts the packet and handed it to the userspace. This mode still taxes the host’s CPU of course, so there is another mode where the kernel offloads th
- The beauty of the CPUIf you are bored of contemporary topics of AI and need a breather, I invite you to join me to explore a mundane, fundamental and earthy topic.The CPU.A reading of my substack article https://hnasr.substack.com/p/the-beauty-of-the-cpu
- Sequential Scans in Postgres just got fasterThis new PostgreSQL 17 feature is game changer. They know can combine IOs when performing sequential scan. Grab my database coursehttps://courses.husseinnasser.com
- Does discipline work?No technical video today, just talking about the idea of discipline and consistency.
- Socket management and Kernel Data structuresFundamentals of Operating Systems Course This video is an overview of how the operating system kernel does socket management and the different data structures it utilizes to achieve that.timestamps0:00 Intro1:38 Socket vs Connections7:50 SYN and Accept Queue18:56 Socket Sharding23:14 Receive and Send buffers27:00 Summary
- The genius of long pollingPolling is the ability to interrogate a backend to see if a piece of information is ready. It can introduce a chatty system and as a result long polling was born. In this video I explain the beauty of this design pattern and how we can push it to its limit. 0:00 Intro 0:45 Polling 2:30 Problem with Polling 3:50 Long Polling 8:18 Timeouts 10:00 Long Polling Benefits 12:00 Make requests into Long Polling 17:36 Request Resumption 21:40 Summary
- Six stages of a good software engineerYou get better as a software engineer when you go through these stages. 0:00 Intro 1:15 Understand a technology 7:07 Articulate how it works 15:30 Understand its’ limitations 19:48 Try to build something better 27:45 Realize what you built also has limitations 32:48 Appreciate the original tech as is Understand a technology We use technologies all the time without knowing how it works. And it is ok not knowing how things work if interests isn’t there. But when there is interest to und
- This new Linux patch can speed up Reading RequestsFundamentals of Operating Systems Course https://oscourse.win Very clever! We often call read/rcv system call to read requests from a connection, this copies data from kernel receive buffer to user space which has a cost. This new patch changes this to allow zero copy with notification. “Reading' data out of a socket instead becomes a “notification” mechanism, where the kernel tells userspace where the data is.” This kernel patch enables zero copy from the receive queue. https://lore.k
- Cloudflare's 150ms global cache purge | Deep DiveCloudflare built a global cache purge system that runs under 150 ms. This is how they did it. Using RockDB to maintain local CDN cache, and a peer-to-peer data center distributed system and clever engineering, they went from 1.5 second purge, down to 150 ms. However, this isn’t full picture, because that 150 ms is just actually the P50. In this video I explore Clouldflare CDN work, how the old core-based centralized quicksilver, lazy purge work compared to the new coreless, decentralized act
- MySQL is having a bumpy journeyFundamentals of Database Engineering udemy course https://databases.win MySQL has been having bumpy journey since 2018 with the release of the version 8.0. Critical crashes that made to the final product, significant performance regressions, and tons of stability and bugs issues. In this video I explore what happened to MySql, are these issues getting fixed? And what is the current state of MySQL at the end of 2024. 0:00 Intro 2:00 MySQL 8.0 vs 5.7 Performance 11:00 Critical Crash in 8.0.38
- How many kernel calls in NodeJS vs Bun vs Python vs native CFundamentals of Operating Systems Course https://oscourse.win In this video I use strace a performance tool that measures how many system calls does a process makes. We compare a simple task of reading from a file, and we run the program in different runtimes, namely nodejs, buns , python and native C. We discuss the cost of kernel mode switches, system calls and pe 0:00 Intro 5:00 Code Explanation 6:30 Python 9:30 NodeJS 12:30 BunJS 13:12 C 16:00 Summary
- When do you use threads?Fundamentals of Operating Systems Course https://os.husseinnasser.com When do you use threads? I would say in scenarios where the task is either 1) IO blocking task 2) CPU heavy 3) Large volume of small tasks In any of the cases above, it is favorable to offload the task to a thread. 1) IO blocking task When you read from or write to disk, depending on how you do it and the kernel interface you used, the write might be blocking. This means the process that executes the IO will not be al
- Frontend and Backends TimeoutsI am fascinated by how timeouts affect backend and frontend programming. When a party is waiting on something you can place a timeout to break the wait. This is useful for freeing resources to more critical processes, detecting slow operations and even avoiding DOS attacks. Contrary to common beliefs, timeouts are not exclusive to request processing, they can be applied to other parts of the frontend-backend communications. Let us explore this briefly. 0:00 Intro 2:30 Connection Timeout 5:00 R
- Postgres is combining IO in version 17Learn more about database and OS internals, check out my courses Fundamentals of database engineering https://databases.win Fundamentals of operating systems https://oscourse.win This new PostgreSQL 17 feature is game changer. You see, postgres like most databases work with fixed size pages. Pretty much everything is in this format, indexes, table data, etc. Those pages are 8K in size, each page will have the rows, or index tuples and a fixed header. The pages are just bytes in files and t
- Windows vs Linux KernelFundamentals of Operating Systems Course https://os.husseinnasser.com Why Windows Kernel connects slower than Linux I explore the behavior of TCP/IP stack in Windows kernel when it receives a RST from the backend server especially when the host is available but the port we are trying to connect to is not. This behavior is exacerbated by having both IPv6 and IPv4 and if the happy eye ball protocol is in place where IPv6 is favorable. 0:00 Intro 0:30 Fundamentals TCP/IP 3:00 Unreachable Port
- Running out of TCP ephemeral source portsIn this episode of the backend engineering show I describe an interesting bug I ran into where the web server ran out of ephemeral ports causing the system to halt. 0:00 Intro 0:30 System architecture 2:20 The behavior of the bug 4:00 Backend Troubleshooting 7:00 The cause 15:30 Ephemeral ports on loopback
- io uring gets even fasterFundamentals of Operating Systems Course https://os.husseinnasser.com Linux I/O expert and subsystem maintainer Jens Axboe has submitted all of the IO_uring feature updates ahead of the imminent Linux 6.10 merge window. In this video I explore this with a focus on what zerocopy. 0:00 Intro 0:30 IO_uring gets faster 2:00 What is io_uring 7:00 How Normal Copying Work 12:00 How Zero Copy Works 13:50 ZeroCopy and TLS https://www.phoronix.com/news/Linux-6.10-IO_uring https://lore.kernel.org/i
- They made Python faster with this compiler optionFundamentals of Operating Systems Course https://oscourse.win Looks like fedora is compiling cpython with the -o3 flag, which does aggressive function inlining among other optimizations. This seems to improve python benchmarks performance by at most 1.16x at a cost of an extra 3MB in binary size (text segment). Although it does seem to slow down some benchmarks as well though not significantly. O1 - local register allocation, subexpression elimination O2 - Function inlining only small funct
- How Apache Kafka got faster by switching ext4 to XFShttps://oscourse.win Allegro improved their Kafka produce tail latency by over 80% when they switched from ext4 to xfs. What I enjoyed most about this article is the detailed analysis and tweaking the team made to ext4 before considering switching to xfs. This is a classic case of how a good tech blog looks like in my opinion. 0:00 Intro 0:30 Summary 2:35 How Kafka Works? 5:00 Producers Writes are Slow 7:10 Tracing Kafka Protocol 12:00 Tracing Kernel System Calls 16:00 Journaled File Systems
- Google Patches Linux kernel with 40% TCP performanceGet my backend course https://backend.win Google submitted a patch to Linux Kernel 6.8 to improve TCP performance by 40%, this is done via rearranging the tcp structures for better cpu cache lines, I explore this here. 0:00 Intro 0:30 Google improves Linux Kernel TCP by 40% 1:40 How CPU Cache Line Works 6:45 Reviewing the Google Patch https://www.phoronix.com/news/Linux-6.8-Networking https://lore.kernel.org/netdev/20231129072756.3684495-1-lixiaoyan@google.com/ Discovering Backend Bottlenec
- Database Torn pages0:00 Intro 2:00 File System Block vs Database Pages 4:00 Torn pages or partial page 7:40 How Oracle Solves torn pages 8:40 MySQL InnoDB Doublewrite buffer 10:45 Postgres Full page writes
- Cloudflare Open sources Pingora (NGINX replacement)Get my backend course https://backend.win Cloudflare has announced they are opening sources Pingora as a networking framework! Big news, let us discuss 0:00 Intro 0:30 Reasons why Cloudflare built Pingora? 3:00 It is a framework! 7:30 What in Pingora? 11:50 Security in Pingora 13:45 Multi-threading in Pingora 21:00 Customization vs Configuration 25:00 Summary https://blog.cloudflare.com/pingora-open-source/?utm_campaign=cf_blog&utm_content=20240228&utm_medium=organic_social&utm_source=tw
- The Internals of MongoDBhttps://backend.win https://databases.win I’m a big believer that database systems share similar core fundamentals at their storage layer and understanding them allows one to compare different DBMS objectively. For example, How documents are stored in MongoDB is no different from how MySQL or PostgreSQL store rows. Everything goes to pages of fixed size and those pages are flushed to disk. Each database define page size differently based on their workload, for example MongoDB default page si
- The Beauty of Programming LanguagesIn this video I explore the type of languages, compiled, garbage collected, interpreted, JIT and more.
- The Danger of Defaults - A PostgreSQL StoryI talk about default values and how PostgreSQL 14 got slower when a default parameter has changed. Mike's blog https://smalldatum.blogspot.com/2024/02/it-wasnt-performance-regression-in.html
- Database Background writingBackground writing is a process that writes dirty pages in shared buffer to the disk (well goes to the OS file cache then get flushed to disk by the OS) I go into this process in this video
- The Cost of Memory FragmentationFragmentation is a very interesting topic to me, especially when it comes to memory. While virtually memory does solve external fragmentation (you can still allocate logically contiguous memory in non-contiguous physical memory) it does however introduce performance delays as we jump all over the physical memory to read what appears to us for example as contiguous array in virtual memory. You see, DDR RAM consists of banks, rows and columns. Each row has around 1024 columns and each column h
- The Real Hidden Cost of a RequestIn this video I explore the hidden costs of sending a request from the frontend to the backend Heard https://medium.com/@hnasr/the-journey-of-a-request-to-the-backend-c3de704de223
- Why create Index blocks writesFundamentals of Database Engineering udemy course (link redirects to udemy with coupon) https://database.husseinnasser.com Why create Index blocks writes In this video I explore how create index, why does it block writes and how create index concurrently work and allow writes. 0:00 Intro 1:28 How Create Index works 4:45 Create Index blocking Writes 5:00 Create Index Concurrently
- The Problems of an HTTP/3 BackendHTTP/3 is getting popular in the cloud scene but before you migrate to HTTP/3 consider its cost. I explore it here. 0:00 Intro HTTP/3 is getting popular 3:40 HTTP/1.1 Cost 5:18 HTTP/2 Cost 6:30 HTTP/3 Cost https://blog.apnic.net/2023/09/25/why-http-3-is-eating-the-world/
- Encrypted Client Hello - The Pros & ConsThe Encrypted Client Hello or ECH is a new RFC that encrypts the TLS client hello to hide sensitive information like the SNI. In this video I go through pros and cons of this new rfc. 0:00 Intro 2:00 SNI 4:00 Client Hello 8:40 Encrypted Client Hello 11:30 Inner Client Hello Encryption 18:00 Client-Facing Outer SNI 21:20 Decrypting Inner Client Hello 23:30 Disadvantages 26:00 Censorship vs Privacy ECH https://blog.cloudflare.com/announcing-encrypted-client-hello/ https://chromestatus.com/fea
- The Journey of a Request to the BackendFrom the frontend through the kernel to the backend processWhen we send a request to a backend most of us focus on the processing aspect of the request which is really just the last step. There is so much more happening before a request is ready to be processed, most of this step happens in the Kernel. I break this into 6 steps, each step can theoretically be executed by a dedicated thread or process. Pretty much all backends, web servers, proxies, frameworks and even databases have to do all th
- They Enabled Postgres Partitioning and their Backend fell apartIn a wonderful blog, Kyle explores the pains he faced managing a Postgres instance for a startup he works for and how enabling partitioning sigintfically created wait events causing the backend and subsequently NGINX to through 500 errors. We discuss this in this video/podcast https://www.kylehailey.com/post/postgres-partition-pains-lockmanager-waits
- WebTransport - A Backend Game ChangerWebTransport is a cutting-edge protocol framework designed to support multiplexed and secure transport over HTTP/2 and HTTP/3. It brings together the best of web and transport technologies, providing an all-in-one solution for real-time, bidirectional communication on the web. Watch full episode (subscribers only) https://spotifyanchor-web.app.link/e/cTSGkq5XuAb