A 2024 benchmark of main Web API frameworks

We’ll be comparing the read performance of 6 Web APIs frameworks, sharing the same OpenAPI contract from realworld app, a medium-like clone, implemented under multiple languages (PHP, Python, Javascript, Java and C#). Updated in 09/2024.

Update April 2024 for PHP: I replaced previous Apache results by new FrankenPHP. Now PHP is back in the game, with huge performance increase thanks to worker mode.

This is not a basic synthetic benchmark, but a real world benchmark with DB data tests, and multiple scenarios. This post may be updated when new versions of frameworks will be released or any suggestions for performance related improvement in below commentary section.

A state of the art of real world benchmarks comparison of Web APIs is difficult to achieve and very time-consuming as it forces mastering each framework. As performance can highly dependent of:

Code implementation, all made by my own
Fine-tuning for each runtime, so I mostly take the default configuration

Now that’s said, let’s fight !

The contenders #

We’ll be using the very last up-to-date stable versions of each framework, and the latest stable version of the runtime.

I give you all source code as well as public OCI artifacts of each project, so you can test it by yourself quickly.

Framework & Source code	Runtime	ORM
Laravel 11 (api / image)	FrankenPHP 8.3	Eloquent
Symfony 7 (api / image)	FrankenPHP 8.3	Doctrine
FastAPI (api / image)	Python 3.12	SQLAlchemy 2.0
NestJS 10 (api / image)	Node 20	Prisma 5
Spring Boot 3.3 (api / image)	Java 21	Hibernate 6
ASP.NET Core 8 (api / image)	.NET 8.0	EF Core 8

Each project are:

Using PostgreSQL
Using the same OpenAPI contract
Fully tested and functional against same Postman collection
Highly tooled with high code quality in mind (static analyzers, formatter, linters, good code coverage, etc.)
Share roughly the same amount of DB datasets, 50 users, 500 articles, 5000 comments, generated by faker-like library for each language
Avoiding N+1 queries with eager loading (normally)
Containerized with Docker, and deployed on a monitored Docker Swarm cluster

The Swarm cluster for testing #

We’ll be running all Web APIs project on a Docker swarm cluster, where each node are composed of 2 dedicated CPUs for stable performance and 8 GB of RAM. I’ll use 4 CCX13 instances from Hetzner.

Traefik will be used as a reverse proxy, load balancing the requests to the replicas of each node.

flowchart TD client((k6)) client -- Port 80 443 --> traefik-01 subgraph manager-01 traefik-01{Traefik SSL} end subgraph worker-01 app-01([Conduit replica 1]) traefik-01 --> app-01 end subgraph worker-02 app-02([Conduit replica 2]) traefik-01 --> app-02 end subgraph storage-01 DB[(PostgreSQL)] app-01 --> DB app-02 --> DB end

The Swarm cluster is fully monitored with Prometheus and Grafana, allowing to get relevant performance result.

Here is the complete terraform swarm bootstrap if you want to reproduce the same setup.

The deployment configuration #

Following is the deployment configuration for each framework, using Docker Swarm stack file.

deploy-laravel.ymlversion: "3.8"

services:
  app:
    image: gitea.okami101.io/conduit/laravel:latest
    environment:
      - APP_KEY=base64:nltxnFb9OaSAr4QcCchy8dG1QXUbc2+2tsXpzN9+ovg=
      - DB_CONNECTION=pgsql
      - DB_HOST=postgres_db
      - DB_USERNAME=okami
      - DB_PASSWORD=okami
      - DB_DATABASE=conduit_laravel
      - JWT_SECRET_KEY=c2b344e1-1a20-47fc-9aef-55b0c0d568a7
    networks:
      - postgres_db
      - traefik_public
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.laravel.entrypoints=websecure
        - traefik.http.services.laravel.loadbalancer.server.port=8000
      replicas: 2
      placement:
        max_replicas_per_node: 1
        constraints:
          - node.labels.run == true

networks:
  postgres_db:
    external: true
  traefik_public:
    external: true

deploy-symfony.ymlversion: "3.8"

services:
  app:
    image: gitea.okami101.io/conduit/symfony:latest
    environment:
      - SERVER_NAME=:80
      - APP_SECRET=ede04f29dd6c8b0e404581d48c36ec73
      - DATABASE_URL=postgresql://okami:okami@postgres_db/conduit_symfony
      - DATABASE_RO_URL=postgresql://okami:okami@postgres_db/conduit_symfony
      - JWT_PASSPHRASE=c2b344e1-1a20-47fc-9aef-55b0c0d568a7
      - FRANKENPHP_CONFIG=worker ./public/index.php
      - APP_RUNTIME=Runtime\FrankenPhpSymfony\Runtime
    networks:
      - postgres_db
      - traefik_public
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.symfony.entrypoints=websecure
        - traefik.http.services.symfony.loadbalancer.server.port=80
      replicas: 2
      placement:
        max_replicas_per_node: 1
        constraints:
          - node.labels.run == true

networks:
  postgres_db:
    external: true
  traefik_public:
    external: true

deploy-fastapi.ymlversion: "3.8"

services:
  app:
    image: gitea.okami101.io/conduit/fastapi:latest
    environment:
      - DB_HOST=postgres_db
      - DB_RO_HOST=postgres_db
      - DB_PORT=5432
      - DB_USERNAME=okami
      - DB_PASSWORD=okami
      - DB_DATABASE=conduit_fastapi
      - JWT_PASSPHRASE=c2b344e1-1a20-47fc-9aef-55b0c0d568a7
    networks:
      - postgres_db
      - traefik_public
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.fastapi.entrypoints=websecure
        - traefik.http.services.fastapi.loadbalancer.server.port=8000
      replicas: 4
      placement:
        max_replicas_per_node: 2
        constraints:
          - node.labels.run == true

networks:
  postgres_db:
    external: true
  traefik_public:
    external: true

deploy-nestjs.ymlversion: "3.8"

services:
  app:
    image: gitea.okami101.io/conduit/nestjs:latest
    environment:
      - NODE_ENV=production
      - DATABASE_URL=postgres://okami:okami@postgres_db/conduit_nestjs
      - JWT_SECRET=c2b344e1-1a20-47fc-9aef-55b0c0d568a7
    networks:
      - postgres_db
      - traefik_public
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.nestjs.entrypoints=websecure
        - traefik.http.services.nestjs.loadbalancer.server.port=3000
      replicas: 2
      placement:
        max_replicas_per_node: 1
        constraints:
          - node.labels.run == true

networks:
  postgres_db:
    external: true
  traefik_public:
    external: true

deploy-spring-boot.ymlversion: "3.8"

services:
  app:
    image: gitea.okami101.io/conduit/spring-boot:latest
    environment:
      - SPRING_PROFILES_ACTIVE=production
      - DB_HOST=postgres_db
      - DB_PORT=5432
      - DB_RO_HOST=postgres_db
      - DB_USERNAME=okami
      - DB_PASSWORD=okami
      - DB_DATABASE=conduit_springboot
      - JWT_SECRET_KEY=YzJiMzQ0ZTEtMWEyMC00N2ZjLTlhZWYtNTViMGMwZDU2OGE3
    networks:
      - postgres_db
      - traefik_public
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.springboot.entrypoints=websecure
        - traefik.http.services.springboot.loadbalancer.server.port=8080
      replicas: 2
      placement:
        max_replicas_per_node: 1
        constraints:
          - node.labels.run == true

networks:
  postgres_db:
    external: true
  traefik_public:
    external: true

deploy-aspnet-core.ymlversion: "3.8"

services:
  app:
    image: gitea.okami101.io/conduit/symfony:latest
    environment:
      - SERVER_NAME=:80
      - APP_SECRET=ede04f29dd6c8b0e404581d48c36ec73
      - DATABASE_DRIVER=pdo_pgsql
      - DATABASE_URL=postgresql://okami:okami@postgres_db/conduit_symfony
      - DATABASE_RO_URL=postgresql://okami:okami@postgres_db/conduit_symfony
      - JWT_PASSPHRASE=c2b344e1-1a20-47fc-9aef-55b0c0d568a7
      - FRANKENPHP_CONFIG=worker ./public/index.php
      - APP_RUNTIME=Runtime\FrankenPhpSymfony\Runtime
    networks:
      - postgres_db
      - traefik_public
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.symfony.entrypoints=websecure
        - traefik.http.services.symfony.loadbalancer.server.port=80
      replicas: 2
      placement:
        max_replicas_per_node: 1
        constraints:
          - node.labels.run == true

networks:
  postgres_db:
    external: true
  traefik_public:
    external: true

Once the Swarm cluster is ready with all proxy, monitoring, database tools initialized, create all associated databases and let’s deploy each project as following :

docker stack deploy laravel -c deploy-laravel.yml
docker stack deploy symfony -c deploy-symfony.yml
docker stack deploy fastapi -c deploy-fastapi.yml
docker stack deploy nestjs -c deploy-nestjs.yml
docker stack deploy spring-boot -c deploy-spring-boot.yml
docker stack deploy aspnet-core -c deploy-aspnet-core.yml

Once deployed you must migrate and seed the database according each project, check associated README for info.

The k6 scenarios #

We’ll be using k6 to run the tests, with constant-arrival-rate executor for progressive load testing, following 2 different scenarios :

Scenario 1 : fetch all articles, following the pagination
Scenario 2 : fetch all articles, calling each single article with slug, fetch associated comments for each article, and fetch profile of each related author

Duration of each scenario is 1 minute, with a 30 seconds graceful for finishing last started iterations. Results with one single test failures, i.e. any response status different from 200 or any response JSON error parsing, are not accepted.

The Iteration creation rate (rate / time unit) will be chosen in order to obtain the highest possible request rate, without any test failures.

Scenario 1 - Database intensive #

The interest of this scenario is to be very database intensive, by fetching all articles, authors, and favorites, following the pagination, with a couple of SQL queries. Note as each code implementation normally use eager loading to avoid N+1 queries, which can have high influence in this test.

import http from "k6/http";
import { check } from "k6";

export const options = {
    scenarios: {
        articles: {
            env: { CONDUIT_URL: '<framework_url>' },
            duration: '1m',
            executor: 'constant-arrival-rate',
            rate: '<rate>',
            timeUnit: '1s',
            preAllocatedVUs: 50,
        },
    },
};

export default function () {
    const apiUrl = `https://${__ENV.CONDUIT_URL}/api`;

    const limit = 10;
    let offset = 0;

    let articles = []

    do {
        const articlesResponse = http.get(`${apiUrl}/articles?limit=${limit}&offset=${offset}`);
        check(articlesResponse, {
            "status is 200": (r) => r.status == 200,
        });

        articles = articlesResponse.json().articles;

        offset += limit;
    }
    while (articles && articles.length >= limit);
}

Here the expected JSON response format:

{
    "articles": [
        {
            "title": "Laboriosam aliquid dolore sed dolore",
            "slug": "laboriosam-aliquid-dolore-sed-dolore",
            "description": "Rerum beatae est enim cum similique.",
            "body": "Voluptas maxime incidunt...",
            "createdAt": "2023-12-23T16:02:03.000000Z",
            "updatedAt": "2023-12-23T16:02:03.000000Z",
            "author": {
                "username": "Devin Swift III",
                "bio": "Nihil impedit totam....",
                "image": "https:\/\/randomuser.me\/api\/portraits\/men\/47.jpg",
                "following": false
            },
            "tagList": [
                "aut",
                "cumque"
            ],
            "favorited": false,
            "favoritesCount": 5
        }
    ],
    //...
    "articlesCount": 500
}

The expected pseudocode SQL queries to build this response:

SELECT * FROM articles LIMIT 10 OFFSET 0;
SELECT count(*) FROM articles;
SELECT * FROM users WHERE id IN (<articles.author_id...>);
SELECT * FROM article_tag WHERE article_id IN (<articles.id...>);
SELECT * FROM favorites WHERE article_id IN (<articles.id...>);

It can highly differ according to each ORM, as few of them can prefer to reduce the queries by using sub select, but it’s a good approximation.

Scenario 2 - Runtime intensive #

The interest of this scenario is to be mainly runtime intensive, by calling each endpoint of the API.

import http from "k6/http";
import { check } from "k6";

export const options = {
    scenarios: {
        articles: {
            env: { CONDUIT_URL: '<framework_url>' },
            duration: '1m',
            executor: 'constant-arrival-rate',
            rate: '<rate>',
            timeUnit: '1s',
            preAllocatedVUs: 50,
        },
    },
};

export default function () {
    const apiUrl = `https://${__ENV.CONDUIT_URL}.sw.okami101.io/api`;

    const limit = 10;
    let offset = 0;

    const tagsResponse = http.get(`${apiUrl}/tags`);
    check(tagsResponse, {
        "status is 200": (r) => r.status == 200,
    });

    let articles = []

    do {
        const articlesResponse = http.get(`${apiUrl}/articles?limit=${limit}&offset=${offset}`);
        check(articlesResponse, {
            "status is 200": (r) => r.status == 200,
        });

        articles = articlesResponse.json().articles;

        for (let i = 0; i < articles.length; i++) {
            const article = articles[i];
            const articleResponse = http.get(`${apiUrl}/articles/${article.slug}`);
            check(articleResponse, {
                "status is 200": (r) => r.status == 200,
            });

            const commentsResponse = http.get(`${apiUrl}/articles/${article.slug}/comments`);
            check(commentsResponse, {
                "status is 200": (r) => r.status == 200,
            });

            const authorsResponse = http.get(`${apiUrl}/profiles/${article.author.username}`);
            check(authorsResponse, {
                "status is 200": (r) => r.status == 200,
            });
        }
        offset += limit;
    }
    while (articles && articles.length >= limit);
}

The results #

Laravel (Octane) #

Laravel Octane will be enabled with FrankenPHP runtime.

Laravel scenario 1 #

Iteration creation rate = 10/s

checks.........................: 100.00% ✓ 22287      ✗ 0
data_received..................: 236 MB  3.7 MB/s
data_sent......................: 1.9 MB  31 kB/s
dropped_iterations.............: 164     2.570365/s
http_req_blocked...............: avg=40.43µs  min=272ns   med=699ns    max=64ms     p(90)=1.11µs   p(95)=1.29µs
http_req_connecting............: avg=2.37µs   min=0s      med=0s       max=9.31ms   p(90)=0s       p(95)=0s
http_req_duration..............: avg=129.05ms min=4.37ms  med=99.97ms  max=338.95ms p(90)=244.5ms  p(95)=254.83ms
  { expected_response:true }...: avg=129.05ms min=4.37ms  med=99.97ms  max=338.95ms p(90)=244.5ms  p(95)=254.83ms
http_req_failed................: 0.00%   ✓ 0          ✗ 22287
http_req_receiving.............: avg=381.18µs min=34.13µs med=297.73µs max=15.25ms  p(90)=588µs    p(95)=780.82µs
http_req_sending...............: avg=112.76µs min=36.13µs med=99.03µs  max=10.11ms  p(90)=154.25µs p(95)=183.11µs
http_req_tls_handshaking.......: avg=34.66µs  min=0s      med=0s       max=29.3ms   p(90)=0s       p(95)=0s
http_req_waiting...............: avg=128.55ms min=4.17ms  med=99.52ms  max=338.41ms p(90)=243.94ms p(95)=254.31ms
http_reqs......................: 22287   349.303183/s
iteration_duration.............: avg=6.62s    min=1.33s   med=6.87s    max=9.38s    p(90)=7.77s    p(95)=7.99s
iterations.....................: 437     6.849082/s
vus............................: 25      min=10       max=50
vus_max........................: 50      min=50       max=50

We are not database limited.

Laravel scenario 2 #

Iteration creation rate = 1/s

checks.........................: 100.00% ✓ 53163      ✗ 0
data_received..................: 120 MB  1.3 MB/s
data_sent......................: 4.4 MB  49 kB/s
dropped_iterations.............: 6       0.066664/s
http_req_blocked...............: avg=17.49µs  min=252ns   med=678ns    max=76.67ms  p(90)=1.09µs   p(95)=1.27µs
http_req_connecting............: avg=732ns    min=0s      med=0s       max=3.15ms   p(90)=0s       p(95)=0s
http_req_duration..............: avg=59.48ms  min=3.41ms  med=49.42ms  max=336.19ms p(90)=121.32ms p(95)=134.81ms
  { expected_response:true }...: avg=59.48ms  min=3.41ms  med=49.42ms  max=336.19ms p(90)=121.32ms p(95)=134.81ms
http_req_failed................: 0.00%   ✓ 0          ✗ 53164
http_req_receiving.............: avg=219.66µs min=23.57µs med=142.94µs max=15.94ms  p(90)=400.62µs p(95)=567.21µs
http_req_sending...............: avg=108.62µs min=31.16µs med=95.84µs  max=13.53ms  p(90)=153.78µs p(95)=181.93µs
http_req_tls_handshaking.......: avg=14.78µs  min=0s      med=0s       max=29.33ms  p(90)=0s       p(95)=0s
http_req_waiting...............: avg=59.15ms  min=3.25ms  med=49.07ms  max=335.49ms p(90)=120.95ms p(95)=134.56ms
http_reqs......................: 53164   590.690863/s
iteration_duration.............: avg=58.57s   min=40.08s  med=1m0s     max=1m19s    p(90)=1m16s    p(95)=1m17s
iterations.....................: 9       0.099997/s
vus............................: 45      min=1        max=50
vus_max........................: 50      min=50       max=50

This is where Laravel Octane really shines, previously we had less than 300 req/s with Apache.

Symfony (FrankenPHP) #

Symfony scenario 1 #

Iteration creation rate = 10/s

checks.........................: 100.00% ✓ 21930      ✗ 0
data_received..................: 198 MB  3.1 MB/s
data_sent......................: 1.9 MB  30 kB/s
dropped_iterations.............: 171     2.679253/s
http_req_blocked...............: avg=40.29µs  min=266ns   med=720ns    max=65.15ms  p(90)=1.15µs   p(95)=1.34µs
http_req_connecting............: avg=2.38µs   min=0s      med=0s       max=6.02ms   p(90)=0s       p(95)=0s
http_req_duration..............: avg=130.84ms min=7.15ms  med=85.37ms  max=337.78ms p(90)=242.21ms p(95)=252.98ms
  { expected_response:true }...: avg=130.84ms min=7.15ms  med=85.37ms  max=337.78ms p(90)=242.21ms p(95)=252.98ms
http_req_failed................: 0.00%   ✓ 0          ✗ 21930
http_req_receiving.............: avg=428.79µs min=28.72µs med=294.3µs  max=26.69ms  p(90)=717.72µs p(95)=1.05ms
http_req_sending...............: avg=116.86µs min=34.94µs med=100.46µs max=16.13ms  p(90)=157.81µs p(95)=185.95µs
http_req_tls_handshaking.......: avg=34.42µs  min=0s      med=0s       max=29.11ms  p(90)=0s       p(95)=0s
http_req_waiting...............: avg=130.3ms  min=6.88ms  med=84.81ms  max=337.45ms p(90)=241.7ms  p(95)=252.54ms
http_reqs......................: 21930   343.602476/s
iteration_duration.............: avg=6.7s     min=1.41s   med=7.02s    max=9.09s    p(90)=7.96s    p(95)=8.17s
iterations.....................: 430     6.737303/s
vus............................: 22      min=10       max=50
vus_max........................: 50      min=50       max=50

We are database limited, performing same as Laravel.

Symfony scenario 2 #

Iteration creation rate = 2/s

checks.........................: 100.00% ✓ 110192      ✗ 0
data_received..................: 195 MB  2.3 MB/s
data_sent......................: 8.7 MB  105 kB/s
dropped_iterations.............: 49      0.588079/s
http_req_blocked...............: avg=9.46µs   min=237ns   med=582ns    max=59.73ms  p(90)=890ns    p(95)=1.03µs
http_req_connecting............: avg=378ns    min=0s      med=0s       max=3.44ms   p(90)=0s       p(95)=0s
http_req_duration..............: avg=27.36ms  min=2.09ms  med=22.91ms  max=534.89ms p(90)=53.87ms  p(95)=61.96ms
  { expected_response:true }...: avg=27.36ms  min=2.09ms  med=22.91ms  max=534.89ms p(90)=53.87ms  p(95)=61.96ms
http_req_failed................: 0.00%   ✓ 0           ✗ 110192
http_req_receiving.............: avg=393.34µs min=20.27µs med=169.55µs max=44.39ms  p(90)=667.69µs p(95)=1.17ms
http_req_sending...............: avg=96.96µs  min=21.03µs med=82.98µs  max=11.97ms  p(90)=133.54µs p(95)=160.59µs
http_req_tls_handshaking.......: avg=8.02µs   min=0s      med=0s       max=58.6ms   p(90)=0s       p(95)=0s
http_req_waiting...............: avg=26.87ms  min=1.57ms  med=22.42ms  max=534.69ms p(90)=53.21ms  p(95)=61.26ms
http_reqs......................: 110192  1322.481752/s
iteration_duration.............: avg=42.8s    min=19.32s  med=46.97s   max=53.14s   p(90)=51.96s   p(95)=52.6s
iterations.....................: 71      0.852115/s
vus............................: 2       min=2         max=50
vus_max........................: 50      min=50        max=50

Huge gap in performance against Laravel Octane here, about twice better ! Without FrankenPHP, we were capping to previously about 300 req/s on Apache…

FastAPI #

As a side note here, uvicorn is limited to 1 CPU core, so I use 2 replicas on each worker to use all CPU cores.

FastAPI scenario 1 #

Iteration creation rate = 15/s

checks.........................: 100.00% ✓ 29835      ✗ 0
data_received..................: 241 MB  3.9 MB/s
data_sent......................: 2.6 MB  42 kB/s
dropped_iterations.............: 315     5.03316/s
http_req_blocked...............: avg=30.38µs  min=227ns   med=651ns    max=52.06ms  p(90)=1.02µs   p(95)=1.18µs
http_req_connecting............: avg=1.98µs   min=0s      med=0s       max=5.76ms   p(90)=0s       p(95)=0s
http_req_duration..............: avg=97.85ms  min=6.31ms  med=83.03ms  max=500.66ms p(90)=192.52ms p(95)=226.66ms
  { expected_response:true }...: avg=97.85ms  min=6.31ms  med=83.03ms  max=500.66ms p(90)=192.52ms p(95)=226.66ms
http_req_failed................: 0.00%   ✓ 0          ✗ 29835
http_req_receiving.............: avg=570.36µs min=28.53µs med=260.32µs max=21.18ms  p(90)=1.23ms   p(95)=1.94ms
http_req_sending...............: avg=112.03µs min=32.86µs med=95.56µs  max=16.17ms  p(90)=150.61µs p(95)=181.13µs
http_req_tls_handshaking.......: avg=26.39µs  min=0s      med=0s       max=31.75ms  p(90)=0s       p(95)=0s
http_req_waiting...............: avg=97.17ms  min=6.14ms  med=82.29ms  max=497.14ms p(90)=191.78ms p(95)=225.93ms
http_reqs......................: 29835   476.712194/s
iteration_duration.............: avg=5.02s    min=1.44s   med=5.12s    max=6.71s    p(90)=5.77s    p(95)=6s
iterations.....................: 585     9.347298/s
vus............................: 21      min=15       max=50
vus_max........................: 50      min=50       max=50

FastAPI outperforms above PHP frameworks in this specific scenario, and database isn’t the bottleneck anymore.

FastAPI scenario 2 #

Iteration creation rate = 2/s

checks.........................: 100.00% ✓ 64223      ✗ 0
data_received..................: 128 MB  1.4 MB/s
data_sent......................: 4.8 MB  53 kB/s
dropped_iterations.............: 71      0.788854/s
http_req_blocked...............: avg=16.09µs  min=242ns   med=634ns    max=71.12ms  p(90)=1µs      p(95)=1.18µs
http_req_connecting............: avg=936ns    min=0s      med=0s       max=11.27ms  p(90)=0s       p(95)=0s
http_req_duration..............: avg=58.19ms  min=4.23ms  med=33.6ms   max=430.25ms p(90)=146.63ms p(95)=177.64ms
  { expected_response:true }...: avg=58.19ms  min=4.23ms  med=33.6ms   max=430.25ms p(90)=146.63ms p(95)=177.64ms
http_req_failed................: 0.00%   ✓ 0          ✗ 64223
http_req_receiving.............: avg=224.16µs min=20.82µs med=120.98µs max=11.92ms  p(90)=456.57µs p(95)=713.27µs
http_req_sending...............: avg=98.3µs   min=28.53µs med=85.53µs  max=15.47ms  p(90)=136.99µs p(95)=162.39µs
http_req_tls_handshaking.......: avg=13.73µs  min=0s      med=0s       max=44.98ms  p(90)=0s       p(95)=0s
http_req_waiting...............: avg=57.87ms  min=4.04ms  med=33.29ms  max=429.5ms  p(90)=146.17ms p(95)=177.16ms
http_reqs......................: 64223   713.557494/s
iteration_duration.............: avg=1m15s    min=1m6s    med=1m15s    max=1m23s    p(90)=1m23s    p(95)=1m23s
iterations.....................: 11      0.122217/s
vus............................: 39      min=2        max=50
vus_max........................: 50      min=50       max=50

FastAPI fall behind Symfony but ahead of Laravel.

NestJS #

Note that we’re using Fastify adapter instead of Express in order to maximize performance.

NestJS scenario 1 #

checks.........................: 100.00% ✓ 46104      ✗ 0
data_received..................: 791 MB  13 MB/s
data_sent......................: 4.4 MB  72 kB/s
dropped_iterations.............: 296     4.79183/s
http_req_blocked...............: avg=22.48µs  min=250ns    med=610ns    max=56.72ms  p(90)=954ns    p(95)=1.1µs
http_req_connecting............: avg=1.12µs   min=0s       med=0s       max=6.32ms   p(90)=0s       p(95)=0s
http_req_duration..............: avg=62.1ms   min=2.94ms   med=52.66ms  max=322.4ms  p(90)=115.17ms p(95)=120.56ms
  { expected_response:true }...: avg=62.1ms   min=2.94ms   med=52.66ms  max=322.4ms  p(90)=115.17ms p(95)=120.56ms
http_req_failed................: 0.00%   ✓ 0          ✗ 46104
http_req_receiving.............: avg=388.24µs min=25.63µs  med=233.32µs max=205.79ms p(90)=555.24µs p(95)=925.96µs
http_req_sending...............: avg=119.09µs min=29.63µs  med=92.44µs  max=41.89ms  p(90)=148.13µs p(95)=183.72µs
http_req_tls_handshaking.......: avg=20µs     min=0s       med=0s       max=55.7ms   p(90)=0s       p(95)=0s
http_req_waiting...............: avg=61.59ms  min=0s       med=52.07ms  max=171.12ms p(90)=114.69ms p(95)=119.96ms
http_reqs......................: 46104   746.359903/s
iteration_duration.............: avg=3.21s    min=966.59ms med=3.26s    max=4.4s     p(90)=3.7s     p(95)=3.83s
iterations.....................: 904     14.634508/s
vus............................: 31      min=20       max=50
vus_max........................: 50      min=50       max=50

Far ahead of FastAPI, let’s keep up on scenario 2.

NestJS scenario 2 #

Iteration creation rate = 3/s

checks.........................: 100.00% ✓ 141232      ✗ 0
data_received..................: 651 MB  7.4 MB/s
data_sent......................: 12 MB   131 kB/s
dropped_iterations.............: 89      1.010713/s
http_req_blocked...............: avg=7.74µs   min=200ns   med=577ns   max=64.13ms  p(90)=896ns    p(95)=1.04µs
http_req_connecting............: avg=335ns    min=0s      med=0s      max=6.16ms   p(90)=0s       p(95)=0s
http_req_duration..............: avg=24.88ms  min=1.94ms  med=20.22ms max=300.63ms p(90)=49.61ms  p(95)=62.06ms
  { expected_response:true }...: avg=24.88ms  min=1.94ms  med=20.22ms max=300.63ms p(90)=49.61ms  p(95)=62.06ms
http_req_failed................: 0.00%   ✓ 0           ✗ 141232
http_req_receiving.............: avg=415.42µs min=20.02µs med=172.2µs max=210.76ms p(90)=753.01µs p(95)=1.3ms
http_req_sending...............: avg=94.98µs  min=21.16µs med=78.33µs max=25.47ms  p(90)=129.63µs p(95)=158.24µs
http_req_tls_handshaking.......: avg=6.32µs   min=0s      med=0s      max=32.94ms  p(90)=0s       p(95)=0s
http_req_waiting...............: avg=24.37ms  min=0s      med=19.73ms max=295.3ms  p(90)=48.93ms  p(95)=61.29ms
http_reqs......................: 141232  1603.877186/s
iteration_duration.............: avg=38.96s   min=27.77s  med=40.36s  max=47s      p(90)=45.89s   p(95)=46.29s
iterations.....................: 91      1.033426/s
vus............................: 3       min=3         max=50
vus_max........................: 50      min=50        max=50

Now NestJS exceeds the performance of Symfony and FrankenPHP. The native even loop system is very efficient. Note as it’s configured to use Fastify under the hood instead of Express. With Express it match 1200 req/s, so Fastify really shines here. It’s time to test it against compiled language.

Spring Boot #

Spring Boot scenario 1 #

Iteration creation rate = 40/s

checks.........................: 100.00% ✓ 95676       ✗ 0
data_received..................: 1.8 GB  30 MB/s
data_sent......................: 8.2 MB  135 kB/s
dropped_iterations.............: 525     8.61974/s
http_req_blocked...............: avg=13.02µs min=197ns    med=519ns    max=66.13ms  p(90)=752ns    p(95)=866ns
http_req_connecting............: avg=940ns   min=0s       med=0s       max=7.96ms   p(90)=0s       p(95)=0s
http_req_duration..............: avg=29.35ms min=2.59ms   med=26.27ms  max=184.74ms p(90)=49.8ms   p(95)=59.97ms
  { expected_response:true }...: avg=29.35ms min=2.59ms   med=26.27ms  max=184.74ms p(90)=49.8ms   p(95)=59.97ms
http_req_failed................: 0.00%   ✓ 0           ✗ 95676
http_req_receiving.............: avg=2.31ms  min=24.31µs  med=991.85µs max=87.57ms  p(90)=5.43ms   p(95)=9.12ms
http_req_sending...............: avg=186.7µs min=24.8µs   med=81.4µs   max=56.81ms  p(90)=159.29µs p(95)=243.03µs
http_req_tls_handshaking.......: avg=10.78µs min=0s       med=0s       max=54.77ms  p(90)=0s       p(95)=0s
http_req_waiting...............: avg=26.85ms min=0s       med=23.8ms   max=180.66ms p(90)=46.18ms  p(95)=55.49ms
http_reqs......................: 95676   1570.861508/s
iteration_duration.............: avg=1.55s   min=775.41ms med=1.56s    max=2.12s    p(90)=1.71s    p(95)=1.76s
iterations.....................: 1876    30.801206/s
vus............................: 49      min=33        max=50
vus_max........................: 50      min=50        max=50

End of debate, Spring Boot destroys competition for 1st scenario. Moreover, database is the bottleneck, and java runtime is clearly sleeping here. But JPA Hibernate was difficult to tune for optimal performance, and finally the magic @BatchSize annotation was the key, allowing to merge n+1 queries into 1+1 queries. Without it, Spring Boot was performing 3 times slower !

Spring Boot scenario 2 #

Iteration creation rate = 10/s

checks.........................: 100.00% ✓ 173824      ✗ 0
data_received..................: 774 MB  11 MB/s
data_sent......................: 15 MB   215 kB/s
dropped_iterations.............: 489     6.809786/s
http_req_blocked...............: avg=7.54µs   min=212ns   med=529ns   max=52.17ms  p(90)=742ns    p(95)=852ns
http_req_connecting............: avg=292ns    min=0s      med=0s      max=4.99ms   p(90)=0s       p(95)=0s
http_req_duration..............: avg=17.72ms  min=1.88ms  med=14.67ms max=246.82ms p(90)=33.14ms  p(95)=41.12ms
  { expected_response:true }...: avg=17.72ms  min=1.88ms  med=14.67ms max=246.82ms p(90)=33.14ms  p(95)=41.12ms
http_req_failed................: 0.00%   ✓ 0           ✗ 173824
http_req_receiving.............: avg=2.56ms   min=23.95µs med=1.45ms  max=58.29ms  p(90)=6.13ms   p(95)=8.97ms
http_req_sending...............: avg=102.03µs min=22.45µs med=76.81µs max=42.01ms  p(90)=127.42µs p(95)=163.59µs
http_req_tls_handshaking.......: avg=6.34µs   min=0s      med=0s      max=50.96ms  p(90)=0s       p(95)=0s
http_req_waiting...............: avg=15.06ms  min=0s      med=12ms    max=244.1ms  p(90)=29.02ms  p(95)=36.66ms
http_reqs......................: 173824  2420.663008/s
iteration_duration.............: avg=27.87s   min=12s     med=29.74s  max=31.63s   p(90)=30.76s   p(95)=31.06s
iterations.....................: 112     1.559706/s
vus............................: 12      min=10        max=50
vus_max........................: 50      min=50        max=50

Java is maybe not the best DX experience for me, but it’s a beast in terms of raw performance. Besides, we’ll again have database bottleneck, which is the only case seen in this scenario on every framework tested ! Impossible to reach 100% java runtime CPU usage, even with 4 CPU cores, staying only at 60-70% overall…

ASP.NET Core #

ASP.NET Core scenario 1 #

Iteration creation rate = 20/s

checks.........................: 100.00% ✓ 55590     ✗ 0
data_received..................: 1.3 GB  21 MB/s
data_sent......................: 5.1 MB  83 kB/s
dropped_iterations.............: 110     1.793989/s
http_req_blocked...............: avg=17.61µs min=206ns    med=575ns    max=65.29ms  p(90)=888ns    p(95)=1.03µs
http_req_connecting............: avg=806ns   min=0s       med=0s       max=3.77ms   p(90)=0s       p(95)=0s
http_req_duration..............: avg=48.38ms min=2.81ms   med=43.08ms  max=286.36ms p(90)=94.36ms  p(95)=110.84ms
  { expected_response:true }...: avg=48.38ms min=2.81ms   med=43.08ms  max=286.36ms p(90)=94.36ms  p(95)=110.84ms
http_req_failed................: 0.00%   ✓ 0         ✗ 55590
http_req_receiving.............: avg=1.46ms  min=22.08µs  med=596.35µs max=51.59ms  p(90)=3.1ms    p(95)=5.66ms
http_req_sending...............: avg=141.6µs min=29.36µs  med=88.9µs   max=36.89ms  p(90)=156.02µs p(95)=214.06µs
http_req_tls_handshaking.......: avg=15.21µs min=0s       med=0s       max=34.23ms  p(90)=0s       p(95)=0s
http_req_waiting...............: avg=46.77ms min=0s       med=41.21ms  max=280.31ms p(90)=92.33ms  p(95)=108.67ms
http_reqs......................: 55590   906.61695/s
iteration_duration.............: avg=2.52s   min=647.26ms med=2.6s     max=3.5s     p(90)=2.97s    p(95)=3.05s
iterations.....................: 1090    17.776803/s
vus............................: 19      min=14      max=50
vus_max........................: 50      min=50      max=50

ASP.NET Core is performing well here. EF Core is incredibly efficient by default without any tuning headaches as it was with Sping Boot.

ASP.NET Core scenario 2 #

Iteration creation rate = 10/s

checks.........................: 100.00% ✓ 155200     ✗ 0
data_received..................: 946 MB  13 MB/s
data_sent......................: 14 MB   191 kB/s
dropped_iterations.............: 501     6.896565/s
http_req_blocked...............: avg=7.71µs   min=194ns   med=532ns   max=58.81ms  p(90)=794ns    p(95)=926ns
http_req_connecting............: avg=342ns    min=0s      med=0s      max=8.25ms   p(90)=0s       p(95)=0s
http_req_duration..............: avg=21.65ms  min=1.63ms  med=15.74ms max=322.81ms p(90)=45.46ms  p(95)=58.96ms
  { expected_response:true }...: avg=21.65ms  min=1.63ms  med=15.74ms max=322.81ms p(90)=45.46ms  p(95)=58.96ms
http_req_failed................: 0.00%   ✓ 0          ✗ 155200
http_req_receiving.............: avg=1.78ms   min=19.8µs  med=848.9µs max=68.46ms  p(90)=4.39ms   p(95)=6.92ms
http_req_sending...............: avg=105.11µs min=17.95µs med=80.74µs max=44.98ms  p(90)=133.41µs p(95)=169.13µs
http_req_tls_handshaking.......: avg=6.35µs   min=0s      med=0s      max=37.09ms  p(90)=0s       p(95)=0s
http_req_waiting...............: avg=19.76ms  min=0s      med=13.7ms  max=320.85ms p(90)=42.87ms  p(95)=56.35ms
http_reqs......................: 155200  2136.42093/s
iteration_duration.............: avg=33.97s   min=29.96s  med=34.26s  max=36.95s   p(90)=35.59s   p(95)=35.78s
iterations.....................: 100     1.37656/s
vus............................: 15      min=10       max=50
vus_max........................: 50      min=50       max=50

Not that far to Java variant, just a bit behind. But as workers are fully loaded here, contrary to Spring Boot which is limited by database, Java stays by far the clear winner for raw performance (in sacrifice of some memory obviously).

Conclusion #

Here are the final req/s results for each framework against PgSQL database.

To resume, compiled languages have always a clear advantage when it comes to raw performance.

Performance isn’t the main criteria for a web framework. The DX is also very important, in that regard Laravel stays a very nice candidate, and you always have Octane for high performance if needed, while it’s far less behind than Symfony.

As we have seen with Symfony, PHP is now really back in the game in terms of raw performance, almost competing against NodeJS and outperforming Python. And no more any headaches for worker configuration thanks to the excellent FrankenPHP runtime which provides production optimized docker images.

When it comes to compiled languages, I still personally prefer the DX of ASP.NET Core over Spring Boot. The performance gap is negligible, and it hasn’t this warm up Java feeling and keeps a reasonable memory footprint.