API Design

· by

Contents

Today, I gave API design a little bit of thought. Web APIs to be more specific. So here are some good practices I've seen.

General Thoughts

Once others use your API, every change can break their software. For a public API, you don't even know who they are. So you have to keep things running for quite a while, and changes need a plan (see Versioning). You want your stuff to be reliable, so think about that in advance.

Oh, and JSON is likely the most common way to return information on the web. So make sure your responses are valid JSON and you include Content-Type: application/json in the header. Don't assume that requests are valid JSON, though: validate every payload. Answer with 400 Bad Request if it is broken and with 415 Unsupported Media Type if the client sends something else.

Character Sets

Use UTF-8. Always. Everywhere.

See also: Should character encodings besides UTF-8 (and maybe UTF-16/UTF-32) be deprecated?

Monitoring

You offer a service, so please make sure it is available. You could use services like Runscope which call your endpoints regularly.

To give your users a way to check if things look normal, you could create a status page. Popular choices are statuspage.io and Atlassian Statuspage. Some examples of status pages:

Server Time

Make sure your server time is not too far off. This makes comparing logs of different systems easier. And it could be that you use the system time directly or indirectly as an API output. You can use an NTP server to sync the time.

For manual checking: on current Linux systems, timedatectl shows System clock synchronized: yes if the clock is synced. If you run ntpd, ntpq -pn shows the details:

$ ntpq -pn
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.ubuntu.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 1.ubuntu.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 2.ubuntu.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 3.ubuntu.pool.n .POOL.          16 p    -   64    0    0.000    0.000   0.000
 ntp.ubuntu.com  .POOL.          16 p    -   64    0    0.000    0.000   0.000
+92.222.82.98    130.149.17.8     2 u   75  128  377   24.430   -0.043   1.522
*185.51.192.34   46.243.26.34     2 u   44  128  377   27.447   -0.319   1.120
-212.18.3.19     212.18.1.106     2 u   61  128  377   16.443   -0.317   1.454
+88.159.1.196    193.190.230.66   2 u   47  128  377   29.346    0.564   1.828
-192.33.96.102   .PPS.            1 u  113  128  377   40.525    3.008   2.11

Naming

Suppose we have an API which lets us create / retrieve / update / delete users. I would expect the following:

  • GET example.com/api/users: Returns a list of all users
  • GET example.com/api/users/42: Returns the user with ID 42
  • POST example.com/api/users: Creates a new user
  • PUT example.com/api/users/42: Replaces the user with ID 42 (PATCH: changes only the fields that are sent)
  • DELETE example.com/api/users/42: Deletes the user with ID 42

This kind of logic should be consistent through the complete API.

For the field names, pick one style and use it everywhere. JSON itself does not prescribe one: camelCase is common in the JavaScript world, but many large APIs use snake_case, for example created_at in the GitHub REST API.

For the resource names, I suggest using plural forms (/users, not /user). The main point is to be consistent.

And, of course, there is the actual value of IDs. Sequential IDs tell others how many users you have and let them iterate through your data. Random IDs such as UUIDs (version 4) avoid that. But they are no access control: every endpoint still has to check whether the caller may access the requested object. Missing these checks is the first risk in the OWASP API Security Top 10.

Versioning

Having a URL structure like example.com/api/v1/foobar is really nice. This allows you to change some of the practices and be consistent. So you could follow some naming schemes for /v1/* and others for /v2/*. Or change the authentication.

Not every change needs a new version. Adding an endpoint or an optional field is backwards compatible, as long as clients ignore fields they don't know; tell them to do so. Removing or renaming a field, changing its type or meaning, or making a parameter required breaks clients and needs a new version.

When you plan to remove an old version, announce it early. The Deprecation header (RFC 9745) and the Sunset header (RFC 8594) tell clients in every response that an endpoint is deprecated and when it will go away.

List-returning Endpoints

Please note: You might want to read Web API Pagination with the 'Timestamp_Offset_Checksum' Continuation Token - don't take the following as the best option for granted!

Whenever you return lists, make them paginated. Add a default sorting and an offset parameter. You could also add an order and a limit parameter.

The endpoint should return:

{"continue": 100, "data": [...]}

where the value of continue is the offset of the next page. So in case you decide that a limit of 100 returned elements is enough / reasonable, the first query that has more than 100 elements would return the continue of 100. A call with an offset=142 would have a "continue": 242 in the response, if there are more than 242 elements. On the last page, continue is missing.

It is done similarly in the MediaWiki API.

Django REST framework uses this format for its page number and limit/offset pagination, here for ?limit=100&offset=100:

{
    "count": 1023,
    "next": "https://example.com/api/users?limit=100&offset=200",
    "previous": "https://example.com/api/users?limit=100",
    "results": [...]
}

count is the total number of elements. next and previous are complete URLs, so clients don't have to build them; they are null on the last and on the first page. The cursor pagination of Django REST framework returns the same format without count.

Offset pagination has two weaknesses: if elements are inserted or deleted while a client pages through the list, it skips or repeats elements, and large offsets are slow in many databases. A continuation token (see the note above) avoids both.

Documentation

Describe your API in an OpenAPI document. It is the de-facto standard: tools generate interactive documentation, client libraries and tests from it. Give a working example request and response for every endpoint.

A nice addition for humans: MediaWiki has a format=jsonfm parameter that shows the response as a formatted HTML page in the browser. See MediaWiki as an example.

Also, you should be really careful about units:

  • Prices: Never send a price as a bare number. I suggest an object with three fields: {"amount": 1999, "digits": 2, "currency": "EUR"} means 19.99 EUR (1999 / 100). amount is an integer, so you don't have to deal with floating point precision problems. digits says how many decimal places the amount has, and currency is the ISO 4217 code. Because digits is part of the value, clients don't need a table of currencies, and not every currency has 100 subunits: the Japanese yen has none ({"amount": 500, "digits": 0, "currency": "JPY"}), the Kuwaiti dinar has 1000 fils ({"amount": 1250, "digits": 3, "currency": "KWD"} is 1.250 KWD). digits also covers prices that are more precise than the currency, like fuel at 1.799 EUR per liter ({"amount": 1799, "digits": 3, "currency": "EUR"}). Stripe also sends integer amounts, but without digits: clients have to know from its documentation which currencies have no decimal places.
  • Dates: There are timezones. I suggest using RFC 3339, a strict profile of ISO 8601, in UTC: 2018-03-07T21:37:24Z. JavaScript's toISOString() produces this format with milliseconds: 2018-03-07T21:37:24.000Z. For a date without a time, use 2018-03-07. See also my article about datetime.
  • SI Units: Use SI units as a default. If you don't use them, make a big warning sign.

Errors

Help your API users by making nice error messages. Something like:

HTTP Status 400

{
    "errors": [
        {
            "status": 400,
            "error": "FIELDS_VALIDATION_ERROR",
            "detail": "Invalid country 'German'.",
            "fields": {
                "country": "Invalid country. See ISO 3166 ALPHA-3 for valid values."
            },
            "code": 1337,
            "links": {
                "docs": "https://example.com/docs",
                "errors": "https://example.com/docs/errors#1337"
            }
        }
    ]
}

There is a standard for this: Problem Details for HTTP APIs (RFC 9457) with the media type application/problem+json and the fields type, title, status, detail and instance. Unless you have a reason to do otherwise, use it. The example above as Problem Details:

HTTP/1.1 400 Bad Request
Content-Type: application/problem+json

{
    "type": "https://example.com/problems/validation-error",
    "title": "Your request is not valid.",
    "status": 400,
    "detail": "The request contains 1 invalid field.",
    "instance": "urn:uuid:3f6c2a7e-8a4d-4f1b-9c2e-5d1a0b7e4c21",
    "errors": [
        {
            "detail": "Invalid country 'German'. See ISO 3166 ALPHA-3 for valid values.",
            "pointer": "#/country"
        }
    ]
}
  • type is a URI that identifies the kind of problem. Ideally it is a page in your documentation that explains the error and how to fix it.
  • title is a short summary of that kind of problem. It is the same every time this problem occurs.
  • status repeats the HTTP status code.
  • detail explains this particular occurrence.
  • instance identifies this particular occurrence, e.g. with a unique ID. If a user sends it to your support, you find the matching log entry.
  • You may add your own fields. errors with a JSON Pointer to every invalid field is the pattern the RFC itself uses as an example.

Django REST framework uses a simpler format by default. A validation error (400) maps every field to a list of messages; errors that don't belong to a single field go to non_field_errors:

HTTP/1.1 400 Bad Request
Content-Type: application/json
Vary: Accept
Allow: POST, OPTIONS
X-Frame-Options: DENY
Content-Length: 85
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Cross-Origin-Opener-Policy: same-origin

{"email":["This field is required."],"country":["\"German\" is not a valid choice."]}

This is the complete response of Django 6.1 with Django REST framework 3.18 to a POST /api/users with {"name": "Martin", "country": "German"}: email is missing and country must be one of DEU, FRA or USA. Content-Type, Vary and Allow come from Django REST framework, the other headers from the default security middleware of a new Django project.

All other errors have a single detail field, e.g. {"detail": "Authentication credentials were not provided."} or {"detail": "Not found."}. With a custom exception handler (EXCEPTION_HANDLER setting), Django REST framework can return Problem Details instead.

Never put stack traces or other internal details into error messages. Log them on the server.

Security

HTTPS

Use HTTPS. Redirecting plain HTTP to HTTPS does not help much for an API: a client that sent its token over plain HTTP has already exposed it. Better reject plain HTTP requests.

Authentication

Give a concrete example of how one should authenticate. Most likely, there will be a token system. Be absolutely clear when the tokens expire.

Clients should send the token in the Authorization header, never in the URL: URLs end up in server logs, proxy logs and the browser history.

If losing the token could lead to people not being able to access your service for a while, make sure they know it! This could be the case if one account may only have one active token.

See also:

Input Validation

Validate all input: types, lengths, ranges and allowed values. Validation alone does not prevent injections, though. Never build database queries from user input by string concatenation; use parameterized queries.

See SQL injection and my article about SQL injections.

Rate Limiting

Limit how many requests a client may make, so that one client cannot take down the service for everyone. Answer with 429 Too Many Requests (RFC 6585) and a Retry-After header that says when the client may try again.

HTTP Methods and Status Codes

HTTP Methods

Copied from developer.mozilla.org:

  • GET: [...] Requests using GET should only retrieve data.
  • POST: The POST method is used to submit an entity to the specified resource, often causing a change in state or side effects on the server
  • PUT: The PUT method replaces all current representations of the target resource with the request payload.
  • PATCH: The PATCH method is used to apply partial modifications to a resource.
  • DELETE: The DELETE method deletes the specified resource.

One property matters for clients that retry after a timeout: GET, PUT and DELETE are idempotent (RFC 9110), sending the same request twice has the same effect as sending it once. POST and PATCH are not. If a POST times out, the client cannot know whether the user was created. Stripe solves this with an idempotency key: the client sends a unique key with the request, and the server performs a request with a known key only once.

HTTP Response Status codes

Success: 2XX

  • 200 OK: Successful GET, PUT, PATCH
  • 201 Created: A POST created a resource; its URL goes into the Location header
  • 202 Accepted: The request was accepted and will be processed later
  • 204 No Content: Success without a response body, e.g. for a DELETE

Client Errors: 4XX

  • 400 Bad Request: E.g. the parameters are invalid
  • 401 Unauthorized: Despite the name, the client is not authenticated: the token is missing or invalid
  • 403 Forbidden: Authenticated, but the client may not access the resource
  • 404 Not Found
  • 409 Conflict: The request conflicts with the current state, e.g. the user already exists
  • 410 Gone: The resource existed, but was removed for good, e.g. an endpoint after its sunset date
  • 415 Unsupported Media Type: E.g. the body is not JSON
  • 422 Unprocessable Content: The JSON is valid, but the values are not (some APIs use 400 for this)
  • 429 Too Many Requests: The client exceeded the rate limit

Server Errors: 5XX

  • 500 Internal Server Error
  • 503 Service Unavailable: E.g. maintenance or overload; add a Retry-After header

GraphQL

This article was mainly about REST-like APIs. GraphQL is a query language for APIs which Facebook developed internally and published in 2015; today the GraphQL Foundation maintains it. It is an alternative to REST, see GraphQL vs REST.

A GraphQL API has a typed schema that clients can query, so tools can show all types and fields automatically. You still have to document what the fields mean. You buy that advantage with overhead in writing the resolvers, and with new problems: HTTP caching is harder because all queries go to one endpoint, and clients can send very expensive queries, so you need limits for query depth and complexity.

See also