The Startup Handshake

Why this stage exists

Your server answers nc now. Nothing else can talk to it, because every database client speaks some database's own protocol. Here is psql, PostgreSQL's terminal client, pointed at your server from the last stage:

$psql -h 127.0.0.1 -p 5433 -U droid droid
psql: error: connection to server at "127.0.0.1", port 5433 failed: received invalid response to SSL negotiation: W

psql opened with eight bytes asking for TLS. Your server read them as the start of a line and sent back the welcome line, Welcome to droid!, whose first byte is W. From this stage on, your server speaks PostgreSQL's protocol, version 3.0, the one every PostgreSQL driver in every language speaks: libpq, JDBC, psycopg, node-postgres, pgx. Speaking it is a bigger payoff than it looks. Those drivers, and the tools built on them, can then talk to your database. This stage does the first exchange. The next one sends queries.

The line protocol stays behind --wire text. Every test of the last stage passes that flag and keeps passing.

Messages have lengths, not newlines

Text protocols end a message with a newline, which works until a value contains one. PostgreSQL's protocol instead puts the size of each message in front of it. Every message, both ways, is a type byte, then a 4-byte big-endian length, then a body. The length counts itself and the body, not the type byte:

Everything you learned about lines still holds. One read may hold half a message or several. The input buffer and the rule stay the same: handle a message once all of it has arrived, and keep the rest. Only "where does it end" changes. Before, the answer was "at the next \n". Now it is "length bytes after the type".

There is one more thing to check: the length comes from the client. A length of 3 cannot even hold itself, and a length of 10 MB for a startup message is either a bug or an attack. Check it before waiting for the body, then close that connection and nothing else.

The first message is different

The very first message from a client has no type byte, because the client cannot know yet what the server speaks. It is a length, then a 4-byte number, then a body:

If the number is one of the two special codes, the client asks for encryption: TLS (80877103) or GSS (80877104). This track does not encrypt, so the answer is a single byte, N, with no type and no length. The client then sends a new first message on the same connection, which is the StartupMessage. Otherwise the number is a protocol version: 3 in the high 16 bits and 0 in the low, so 196608. The body is a list of parameter names and values, ended by an empty name:

The greeting

A real PostgreSQL would now ask for a password. Droid trusts every client, so it goes straight to the greeting. These are four kinds of message, in an order every driver waits for:

• ParameterStatus messages tell the driver how to talk to you. client_encoding says strings are UTF-8. standard_conforming_strings says a backslash inside quotes is an ordinary character. server_version lets a driver decide which features to use.
• BackendKeyData gives the client a ticket for later. To cancel a long statement, a client opens a second connection and sends this process id and secret key. That is a later stage, but the key is handed out now. In PostgreSQL the process id is the backend process's real id. In droid it is the session number, which plays the same part.
• ReadyForQuery ends every exchange in this protocol, not just the greeting. Its last byte says the session's state: I for idle, T inside a transaction, E inside a failed one. The client sends nothing more until it sees one.

When the first message is wrong

A client that asks for protocol 2.0 or 4.0 gets an ErrorResponse and is disconnected. An ErrorResponse is a list of fields, each a one-letter code and a string, ended by a zero byte. The severity FATAL means the connection is ending. The SQLSTATE 0A000 is PostgreSQL's code for "feature not supported", and the tests read it. A startup message whose parameters do not end with the final zero byte, the empty name after the last pair, is 08P01, a protocol violation. Bad lengths get no reply at all, because nothing about the message can be trusted. In every case it is only that connection that ends. A server that crashes, or stops serving others, because of one client's bad bytes is broken whatever the client did.

Output contract for this stage

These are the exact messages the tests check. The full definition is on the output contract page, section 94.

What you should see when done

psql gets a step further than before. It connects, reads the greeting and shows its prompt. Answering what you type there is the next stage's work:

$psql -h 127.0.0.1 -p 5433 -U droid droid
psql (16.4, server 16.0)
Type "help" for help.
 
droid=> 

And the kit:

$python3 run.py --stage pg-startup --bin ./c-db
PASS server/server-listen:listens-and-greets
...
PASS server/server-listen:shutdown
PASS server/pg-startup:greeting-sequence
PASS server/pg-startup:golden-bytes
PASS server/pg-startup:ssl-request-refused
PASS server/pg-startup:gss-request-refused
PASS server/pg-startup:byte-at-a-time
PASS server/pg-startup:unknown-protocol-rejected
PASS server/pg-startup:bad-startup-packets
PASS server/pg-startup:terminate-closes
PASS server/pg-startup:wire-pg-flag
================================================================================
summary: passed=23 failed=0 total=23

Common bugs in this stage

• Little-endian lengths. Every integer in this protocol is big-endian: the most significant byte first. On x86 and ARM a plain memcpy of an int gives the wrong order.
• Counting the type byte in the length. The length counts itself and the body. AuthenticationOk is 9 bytes on the wire with a length of 8.
• Sending N with a type and a length. The answer to an SSLRequest is one bare byte.
• Waiting for a 10 MB body, or for 8 bytes that never come. Check the length as soon as its 4 bytes arrive: a length of 3 must close the connection before anything more is sent.
• Checking only the last byte. user\0droid\0 ends with a zero and is still broken: the list needs one more zero, the empty name, after its last pair.
• Reading the first message as a typed message. Only the messages after the startup start with a type byte. A connection needs a flag saying whether its startup is done.
• A fixed process id. Every connection's BackendKeyData carries its own session number. Two clients with the same id could not cancel each other's statements later.

Language notes

The page above is the same for every language. These are the only language-specific hints.

• C: read a big-endian length with ntohl on four bytes copied out of the buffer (never by casting a pointer into the buffer: it may not be aligned). Write one with htonl. A small put_u32 / put_cstr pair that appends to an output buffer keeps the message builders short.
• C++: the same ntohl and htonl, or build the value from the bytes with shifts. A std::string output buffer and a helper that writes the type, leaves 4 bytes, appends the body, then fills in the length, makes every message builder three lines long.
• Rust: u32::from_be_bytes(buf[0..4].try_into()?) and n.to_be_bytes(). A Vec<u8> output buffer, and extend_from_slice.
• Zig: std.mem.readInt(u32, buf[0..4], .big) and std.mem.writeInt(u32, &bytes, n, .big).