Launch by Lunch

Introducing pg-java, a new PostgreSQL driver for the JVM

2026-07-29T00:00:00-04:00

Background
Why a new driver
Things you won't find in other Java drivers
Performance
About that "written by AI" part
A long time coming
Status and what's next
Final Thoughts

Background

I've been working on pgjdbc, the PostgreSQL JDBC driver, for many years. It's a great driver, gets millions of monthly downloads, and it's not going anywhere.

It's also a driver whose shape was decided a very long time ago. JDBC came first and PostgreSQL came second. The JVM it was designed for had no virtual threads, no records, and no sealed types. Most of what you'd want to change about that today can't be changed in place. Doing so would break the applications that depend on the current behavior, and that's most of the Java world talking to PostgreSQL.

So I started a new one: pg-java.

It's a modern, PostgreSQL-specific driver for the JVM, and it is pre-release. But the core driver works, there's a JDBC layer on top of it, and it's been tested far more thoroughly than "pre-release" usually implies.

Why a new driver

Four things drove the design.

PostgreSQL-first. The native API is designed around PostgreSQL's wire protocol and feature set. It is not designed around the lowest common denominator that JDBC has to support across every database on earth. If PostgreSQL can do it, the API should be able to say it directly.

JDBC as a layer, not a foundation. Full JDBC compliance is a long-term goal. There's already a java.sql.* layer that registers a Driver and gives you Connection, PreparedStatement, ResultSet, DatabaseMetaData, DataSource, and XA. But it's built on top of the native API rather than dictating its shape. That ordering matters. Once JDBC's assumptions get into the execution core they never come back out.

Virtual threads for I/O. For a decade, "fast driver" implied an async or reactive API, because the alternative was a thread per connection. On Java 21 that trade is gone. pg-java is ordinary blocking-style code that is careful never to pin a carrier thread (which mostly means ReentrantLock instead of synchronized around I/O). You can run thousands of connections on virtual threads without an event loop or a callback API in sight. It's also significantly easier to reason about both how it works and how you would use it.

Streaming by default. The core query primitive is a pull cursor. It reads exactly enough off the wire to produce the next row and never buffers a whole result set. forEach, map, collect, and friends are adapters built on top of it, not a second read path.

Here's the native API:

PgConnectionConfig config = PgConnectionConfig.builder()
        .host("localhost")
        .database("appdb")
        .user("app")
        .password(secret)
        .build();

try (PgConnection connection = PgConnections.connect(config);
        PgResultStream rows = connection.execute(
                "select id, name from widget where kind = $1", List.of("gear"))) {
    while (rows.next()) {
        Row row = rows.currentRow();
        System.out.println(row.getLong(1) + " " + row.getString(2));
    }
}

Note the $1 and the 1-based column indexes. This is PostgreSQL's protocol, spelled the way PostgreSQL spells it, with no java.sql anywhere.

And here's the same thing through JDBC, which is what most people will actually use:

String url = "jdbc:pg://localhost:5432/appdb?user=app&sslmode=verify-full";
try (Connection connection = DriverManager.getConnection(url, "app", secret);
        PreparedStatement statement =
                connection.prepareStatement("select id from t where name = ?")) {
    statement.setString(1, "widget");
    try (ResultSet rows = statement.executeQuery()) {
        while (rows.next()) {
            System.out.println(rows.getLong(1));
        }
    }
}

The driver owns the jdbc:pg: scheme and also answers to jdbc:postgresql: for ported applications.

There's a third module too: a pgjdbc source-compatibility layer that exposes org.postgresql.Driver and the usual org.postgresql.* vendor types. It's there for applications that want to try the new driver without touching their code. Its target is a migration aid rather than a certified drop-in (and like everything else, still pre-release).

Things you won't find in other Java drivers

Some of what follows is just modern-rewrite table stakes. A few pieces I haven't seen anywhere else.

Batch INSERT rewritten to array parameters

Every driver hits the same wall on bulk DML: a batch of N inserts is N round trips. pgjdbc's answer is reWriteBatchedInserts, which collapses them into one multi-row VALUES statement. That works, but it has costs. The SQL text grows with the batch size. It burns N*M scalar parameters, so it has to chunk to stay under the protocol's 65535-parameter cap. And every distinct batch size is a new statement for the server to parse and plan.

pg-java can rewrite the same batch into array parameters instead:

-- your SQL, batched N times:
INSERT INTO t (a, b, c) VALUES (?, ?, ?)

-- rewritten, one execution, three array parameters of length N:
INSERT INTO t (a, b, c) SELECT * FROM unnest(?::int4[], ?::text[], ?::timestamptz[])

The SQL text is now constant regardless of the batch size. That means one prepared statement, planned once, with perfect cache reuse. It uses M parameters (one array per column) instead of N*M, so it never approaches the parameter cap and never needs chunking. And it's a single round trip and a single server-side execution. In the benchmarks it runs about 2x the non-rewritten path and lands slightly ahead of pgjdbc's best batch mode.

It's opt-in via rewriteBatchUsingArrays and, for now, INSERT-only. That restriction is deliberate. For UPDATE and DELETE you can't reconstruct per-element update counts from a single execution, which silently breaks optimistic-locking checks and anything built on them. UPDATE has a sharper problem. If two batch entries target the same row then UPDATE ... FROM unnest(...) applies one arbitrarily chosen source row instead of applying them in order. The final state of your data can differ, not just the counts. Silently changing what a batch does is not an acceptable price for making it fast.

The pgjdbc-style multi-row VALUES rewrite is implemented too, separately, so the compat layer can reproduce pgjdbc's exact behavior when an application asks for it.

A real pipelining API

The extended query protocol never required waiting for one statement's results before sending the next. libpq has had pipeline mode for years. Java drivers generally don't expose it. pg-java does:

try (PgPipeline p = connection.pipeline()) {
    PgPipelineStatement a = p.queue("insert into t (v) values ($1)", List.of(1));
    PgPipelineStatement b = p.queue("select v from t");
    p.sync();                                  // explicit error boundary
    a.results().summary();                     // results come back in queue order
    try (PgResultStream rs = b.results()) { /* ... */ }
}

sync() is the public error-boundary primitive. Each one closes a section. If the server errors inside a section, the failing handle's read throws, later handles in that same section throw a "skipped" exception carrying the first error as the cause, and later sections are unaffected. Handles are strict FIFO.

The window is byte-budgeted with explicit flow control. That part is not tuning. A pipeline that writes blindly while the server streams rows back will deadlock the moment both TCP buffers fill.

It won't hand your password to an unencrypted socket

sslmode defaults to prefer, matching libpq.

Defaulting it to verify-full sounds better in theory but works out worse in practice. It breaks every server with a self-signed or private-CA certificate, which is most PostgreSQL installations. So it would be a default that everybody has to override, and they'd override it with sslmode=disable. That's strictly worse.

So the credential is gated on encryption instead, and the transport default is left alone. When the server asks for password or md5 authentication over a TCP connection with no TLS, the driver refuses before the credential is disclosed. SCRAM is unaffected, since no password crosses the wire there. Unix sockets are exempt. sslmode itself is untouched.

The check can't live in sslmode. The server chooses the authentication method, or an active interceptor answering in its place does. The disclosure has to be judged where it happens rather than from the policy the client set at connect time.

There's an opt-out, allowUnencryptedPasswordAuth. You'll need it for pre-14 servers using password authentication, since password_encryption defaulted to md5 until then, and for PgBouncer. The pgjdbc compat layer sets it by default. A drop-in that changes your application's security behavior on a jar swap is the one thing a drop-in must never do.

There's also require_auth for accepting or refusing specific authentication methods, and channelBinding=require. Together those give a deployment real downgrade protection.

These defaults are strictly for the core PostgreSQL-specific native driver. The compatibility layer atop it applies the same defaults as pgjdbc.

The public API surface is a file that the build enforces

docs/api-surface.md enumerates every public type in the core module along with its stability tier. A test reflects over the exported packages and fails the build if a public type isn't listed, if a listed type has vanished, or if an @Experimental annotation and the table disagree.

Adding a public type is therefore a deliberate act with a paper trail. It doesn't happen just because some class needed to be visible to a sibling package. The table is executable, not decorative.

Small other stuff

Zero runtime dependencies in the protocol module. It compiles against an empty classpath.
The core module adds only the ongres SCRAM jars and the SLF4J API.
.pgpass support, including libpq's refusal to read a password file that's group or world readable. Reading it is opt-in via passfile or usepassfile, as a driver has no business reading files in your home directory unless you asked it to.
The PG* environment variables are honored natively and ignored by the compat layer, since pgjdbc never read them.
GraalVM native-image metadata, with a smoke test that actually builds and runs a native binary so the metadata can't silently rot.
Server errors surfaced with full fidelity: SQLSTATE, severity, detail, hint, position, schema, table, column, and constraint. Nothing gets flattened into a message string. Even the errors are PostgreSQL at its purest.

Performance

The benchmark harness is driver-agnostic and measures the driver, not the database. The driver under test is supplied at runtime and never bundled.

Against pgjdbc 42.7.13 on PostgreSQL 16, pg-java's native path runs at roughly 101-108% of pgjdbc's throughput across the whole workload sweep. That's select-1, select-by-id, select-rows, prepared, insert, update, returning, types, copy, and mixed. The array batch rewrite roughly doubles batch-insert throughput and edges past pgjdbc's own rewrite.

Allocation is the axis I'd still call unfinished. Some paths are at parity or better. On wide rows both the native and compat layers allocate 26% less than pgjdbc. Others, the write side in particular, allocate substantially more.

The numbers all live in docs/benchmarks/ as dated, append-only reports. Fifteen macro runs so far, including the regressions and the negative results. Run 011 measured a change that turned out to do nothing at all and it's written up anyway. The point of keeping the record is that a failed idea doesn't get re-tried three weeks later.

About that "written by AI" part

Essentially the whole driver was written by Claude, in a loop, over about six weeks. That's roughly 1,400 commits between June 19th and today: about 52,000 lines of main Java, 43,000 lines of tests, 22 architecture decision records, and 70 design documents. The designs were manually reviewed and revised, but the code is Claude's on all but a handful of commits.

I want to be precise about what that means. "AI wrote it" is doing a lot of work in that sentence and it's not the interesting part. The interesting part is the harness around it.

None of which is to say nobody was driving. What I spent six weeks doing was writing the contract and rejecting the wrong answers. AGENTS.md is twelve kilobytes of conventions, module rules, and definitions of done, and it is the single reason that a session with no memory of the previous one still lands code that fits the code already there. The Architecture Decision Records (ADRs) exist for the same reason.

The calls that mattered most were about what not to keep. Routing the multi-row VALUES batch chunks through the general pipelining API looked like obvious reuse, but it measured 61% of pgjdbc's throughput, so it lost to a batch-only primitive built on the same underlying engine (ADR-0021). A security audit flagged the sslmode=prefer default and proposed flipping the transport to verify-full, which I overruled, because the finding was really about the credential and the credential is what should be gated (ADR-0009). Both of those are in the repo with the reasoning attached, which is the part I'd defend. The decisions are auditable rather than vibes.

There is a plan, and the plan is the unit of work. docs/plans/overall.md is a phased, numbered implementation plan where each item (C8.4, N5.7, P4) is intended to be exactly one commit. The loop picks up the next item, implements it, and ticks the checkbox in a separate commit. Anything deferred goes into docs/follow-up.md with the files, the line hints, and the reason, so that a fresh session with no memory of the conversation can pick it up later.

Decisions get written down before they're implemented. Twenty-two ADRs cover the I/O model, module boundaries, the result model, exceptions, TLS posture, secret handling, both batch rewrites, pipelining, and API stability. The ADR index carries two separate columns, Status and Built, because an accepted decision is not a claim that the code does it yet. An agent that would otherwise re-litigate async versus blocking every third session instead reads ADR-0001 and gets on with it.

The build is the fitness function. mvn verify fails on Spotless formatting, a 7-bit-ASCII policy over all source and docs, a module-layering test, a JaCoCo floor, dependency convergence, and the API surface manifest described above. None of those are things a model reliably remembers. All of them are things a build can enforce every single time.

Regression testing is most of the work. There is a full suite of unit tests that run entirely in memory, and an equally wide suite of integration tests on top of that. The unit tests need no Docker at all. There's a MockServer that lets a test drive real protocol byte sequences, so most protocol behavior is testable without a server anywhere in the picture. The integration tests run under Testcontainers across a matrix of server versions (9.1 through 18 are supported and 14 through 18 gate every change), plus dedicated harnesses for TLS, PgBouncer, authentication methods, and Unix sockets.

Baking the container startup into the test suite greatly simplifies the agentic dev loop as targeting a different PostgreSQL server version is a java system property (-Dpg.it.image=...). This allows automated testing of multiple versions in a simple loop.

And then the part I'd recommend to anybody doing this: run somebody else's test suite.

compat-suites/ runs pgjdbc's own upstream test suite, and Hibernate's, against pg-java's compat layer in Docker, and gates on committed baselines. As of the current baseline, 6,851 of pgjdbc's 7,331 test cases pass against a driver that shares no code with it.

Of the remaining tests, they're not all true failures either. The vast majority are tests that cover purely internal aspects of pgjdbc. The compatibility suite does cover some of them, but the primary focus is the public surface area of the driver and matching the behavior of pgjdbc.

A new failure fails the gate. An improvement prints a nudge to refresh the baseline so the win gets locked in. Nothing an agent writes about its own code is worth as much as several thousand tests written by people who were trying to pin down a different implementation's behavior.

So does it work? Mostly, and the failure modes are also interesting.

The loop is very good at breadth. Every codec, every metadata method, every negative TLS case. It's also good at the kind of grinding consistency that humans are bad at, which is most of what a driver's test suite is.

It is prone to declaring victory. That's why the gates and the baselines exist, and why the contributor doc has a rule that says to report honestly and state plainly when a step was skipped.

A long time coming

What's surreal about this entire development process is that both the plan and the result are what we had discussed almost eight years ago:

Date: Thu, 25 Oct 2018 10:42:13 -0400
Subject: Re: Rewriting the driver
From: Sehrope Sarkuni <sehrope@jackdb.com>

Regarding the overall idea, I'm all for a true "next gen" driver though I'd
hope that it'll be something that allows for a more PostgreSQL-specific
backend.

The JDBC layer should be atop of that, not the other way around. For
example, processing rows one by one to limit memory usage shouldn't require
obscure combos of opening transactions and setting fetch sizes.
Notifications and arbitrary response handling should be baked in from the
beginning (ex: I should be able to run "SELECT 1" or "COPY (SELECT 1) TO
STDOUT" without knowing anything in advance about the response type of the
SQL command) .

[ ... truncated ... ]

Two points on actual development of this. It's my experience that things
like don't come together from group discussion. Someone (Dave? :D) needs to
create the "core" that others can then join onto. It's no small undertaking
but it's how all projects of this size come about.

Secondly, and IMHO more importantly, there needs to be a decision from the
beginning of whether backwards compatibility is going to be a design goal.
I'd vote "no" on that. This would be a great chance to clean up the driver
properties and drop any excess baggage. As an example, there could be a
unified track for configuring SSL.

If backwards compatibility is desired, at the very least the core should be
able to stand on its own with any compatibility layer being handle in the
JDBC component.

[ ... truncated ... ]

Regards,
-- Sehrope Sarkuni
Founder & CEO | JackDB, Inc. | https://www.jackdb.com/

Read against what shipped, it's very nearly a checklist.

Processing rows one by one without the obscure combo of an open transaction and a fetch size is the pull cursor. "The core should be able to stand on its own with any compatibility layer being handled in the JDBC component" is the module split, almost word for word. The unified track for configuring SSL is sslmode. Arbitrary response handling landed too: the native API hands you a result stream without needing to know in advance what the command returns.

The one line I keep coming back to is the other one. Someone needs to create the core that others can join onto, and it's no small undertaking. That was true in 2018, and it's why the idea sat for eight years.

It turns out the answer to "who is going to write it" was a model in a loop for six weeks.

Status and what's next

Where it stands today:

Pre-release. Nothing is published to Maven Central yet.
No compatibility guarantees. The API is still moving and will keep moving for a while.
The compat layer is a migration aid, not a certified drop-in. Some PGConnection methods are still stubs.
Two accepted ADRs are unbuilt. JSON value binding and composite-to-record mapping are decided designs that nobody has written yet.

What already works:

Connects over TCP, TLS, and Unix domain sockets.
Authenticates with SCRAM-SHA-256, including channel binding.
Simple and extended query protocols, with a server-side statement cache.
COPY in and out, LISTEN/NOTIFY, and large objects.
Pipelining, and batches with either rewrite strategy.
A JDBC layer with DataSource and XA sitting on top of all of it.
Passes the vast majority of the pgjdbc test suite

To build it you need Java 21, plus Docker if you want to run the integration tests:

git clone https://github.com/pgjdbc/pg-java.git
cd pg-java
./mvnw clean install
./mvnw verify -Pintegration-tests   # needs a Docker daemon

It's released under the PostgreSQL License.

Final Thoughts

If you try it, I want to hear what breaks. That goes double for anyone with an application weird enough to have found the corners of pgjdbc, because those corners are exactly what a new driver gets wrong.

Open an issue on GitHub or email me. If you think you've found a security issue, please use GitHub's private vulnerability reporting instead of a public issue.

Getting the driver this far has been the most interesting open source project I've worked on in years. The next part, where other people start using it and telling me where it's wrong, is the part that actually makes it a driver.

Postgres password encryption without leaking credentials

2024-01-16T00:00:00-05:00

Background
Password Encryption (i.e. "hashing")
PostgreSQL Password Encryption - MD5
PostgreSQL Password Encryption - SCRAM-SHA-256
CREATE USER / ALTER USER
Libraries
Password Generation
Final Thoughts

Background

"Dance like no one is watching. Encrypt like everyone is."

-- Adrian Lamo

The simplest way to create a user to access a PostgreSQL database is something like this:

CREATE USER app PASSWORD 't0pSecret!'

However, the text of this SQL will be sent to the remote database server for execution. That means the plaintext of the password is itself sent to the database.

If statement logging is enabled the following will appear in server logs:

2024-01-14 19:09:09.064 UTC [75] LOG:  statement: CREATE USER app PASSWORD 't0pSecret!';

If there's a pool or proxy in between you and the database server, it could end up in those logs as well.

We can do better. Much better!

Password Encryption (i.e. "hashing")

The PostgreSQL docs refer this topic as "password encryption". In reality, the operation that is being performed is form of hashing and HMAC.

The server does not need to know the exact text of your database user's password. It just needs to know if your supplied credentials match the ones that are authorized to access that user.

Password hashing is one way to perform such a calculation and allows the server to only store the result of that one-way hash function. The server can then check if the result of applying the password hashing operation for the credentials supplied in your connection attempt matches the expected password hash.

PostgreSQL Password Encryption - MD5

Up until PostgreSQL v10, there was only one encryption option built-in to PostgreSQL for dealing with passwords: md5

With md5 password encryption, the server stores MD5(password || username) in its internal pg_shadow catalog (think /etc/shadow but for database passwords).

The hashing operations described in this post use the following expressions:

MD5( ...) = result of md5 hash function of its one and only argument

|| = binary operation to concatenate the raw bytes of its arguments

So 'foo' || 'bar' = 'foobar' and MD5('test') = '098f6bcd4621d373cade4e832627b4f6'

During the authentication step of the PostgreSQL wire protocol, the server picks a random salt and asks the authenticating client to calculate: MD5( MD5(password || username) || salt)).

The random salt prevents the value from a previous connection from being re-used for a later connection attempt (also known as a replay attack). Unfortunately, the size of the salt in the wire protocol is only four-bytes so it's not particularly effective. With enough connection attempts, assuming there is no transport level encryption (TLS), then an eavesdropper could collect every possible authentication response.

PostgreSQL Password Encryption - SCRAM-SHA-256

Starting in PostgreSQL v10, a new option was added and made the default for handling passwords: SCRAM-SHA-256

SCRAM is an acronym for Salted Challenge Response Authentication Mechanism.

It can be used with various hashing algorithms. The PostgreSQL implementation supports only SHA-256.

SCRAM has a number of advantages over md5 encryption:

A modern hash function (SHA-256)
Larger salt sizes (defaulting to 16-bytes)
key stretching (defaulting to 4096 rounds)
Proving to the client that the server knows the user's credentials (the client authenticates the server too!)

With the addition of SCRAM-SHA-256, there is no reason to ever use md5 authentication on a modern PostgreSQL server.

CREATE USER / ALTER USER

The PostgreSQL SQL commands for creating a user, CREATE USER ..., accepts a ... PASSWORD 'literal-value-goes-here' clause for specifying the password. The ALTER USER ... command operates similarly.

The server checks that new password value and decides if it is already encrypted. If not, the server encrypts it using the connection's default encryption method (SHOW password_encryption to see the default).

If the value of the new password starts with md5 then the server assumes it is an already encrypted password using the md5 scheme.

If the value of the new password starts with SCRAM-SHA-256$ then the server assumes it is an already encrypted password using the SCRAM-SHA-256 scheme.

So the following would instruct the PostgreSQL server to encrypt the password using the connections default password encryption:

CREATE USER alice PASSWORD 'abcd'

Whereas this would be treated as an already encrypted md5 password:

CREATE USER alice PASSWORD 'md5fb592cb4152e2aacaaf452714d283f7e'

And this would be treated as an already encrypted SCRAM-SHA-256 password:

CREATE USER alice PASSWORD 'SCRAM-SHA-256$4096:jHhJMplyRcr1io3v3YwabQ==$WNwkp8PFet/L8UUtFBEa7Sn8xSofpPZ4klcfuB0w6Yk=:Oqnfxd3FhyYTALxW+YeU/yVAzxKT+ho08E+hE/'

In all three cases, the user will be able to log in with the same password, abcd, however in the first case the user's password may have been compromised by being logged during the command execution.

The same update could be done via an ALTER USER ... command:

-- Have the server encrypt the password ... BAD
ALTER USER alice PASSWORD 'abcd';

-- Provide an md5 encrypted password ... BETTER (but not great)
CREATE USER alice PASSWORD 'md5fb592cb4152e2aacaaf452714d283f7e'

-- Provide a SCRAM-SHA-256 encrypted password ... BEST (really this is only one you should be using)
CREATE USER alice PASSWORD 'SCRAM-SHA-256$4096:jHhJMplyRcr1io3v3YwabQ==$WNwkp8PFet/L8UUtFBEa7Sn8xSofpPZ4klcfuB0w6Yk=:Oqnfxd3FhyYTALxW+YeU/yVAzxKT+ho08E+hE/'

There's no reason to present the plaintext password to the server. The server does not even save the plaintext of the password. It immediately encrypts it (using the current password_encryption option) and saves the encrypted value:

postgres=# ALTER USER alice PASSWORD 'abcd';
ALTER ROLE

postgres=# SELECT passwd FROM pg_shadow WHERE usename = 'alice';
                                                                passwd                                                                 
---------------------------------------------------------------------------------------------------------------------------------------
 SCRAM-SHA-256$4096:awluLTEXFg1qnYqxpmRNkQ==$FPHOxeG4sUCa+vy6W83VMHaXr3rSRq2QVfkvmAMad2A=:g9oXk/UT6fF+N8z5aFIQpRvLfm8Yz13ZEjdhrWhY2/0=
(1 row)

At best the password is not leaked to some log file. But you'll never really know, right?

Libraries

As creating users and updating their passwords is a common enough task, it's been added to a number of PostgreSQL libraries. The helpers in those libraries handle encrypting the password so that it can be used in SQL commands without exposing its plaintext.

C - libpq

https://www.postgresql.org/docs/current/libpq-misc.html#LIBPQ-PQENCRYPTPASSWORDCONN

The libpq C library is part of core PostgreSQL and is the basis for a number of user tools and other drivers. It provides a function, PQencryptPasswordConn(...) that will encrypt a password using the defaults for the provided connection.

Java - PostgreSQL JDBC Driver (PGJDBC)

https://github.com/pgjdbc/pgjdbc/blob/master/pgjdbc/src/main/java/org/postgresql/util/PasswordUtil.java

A helper class, PasswordUtil was recently added to the PostgreSQL Java database driver, PGJDBC. The methods in that class allow for fine grained control of the encryption type and internal parameters (e.g. number of SCRAM iterations or salt size). There are also methods for generating the ALTER USER ... SQL command for external execution or inclusion in a script.

A new helper method was also added PGConnection that performs a password change operation for a user using the default encryption setting of the database server. This is simplest method for updating a user's password without leaking it along the way.

Only ALTER USER ... PASSWORD ... is supported as creating a user has many additional options and could change over time.

To use it:

String username = "alice";
String newPassword = "my-new-secret-password";
PGConnection pgConn = conn.unwrap(PGConnection.class);
pgConn.alterUserPassword(username, newPassword.toCharArray(), null);

When creating users, the recommended approach is to issue the CREATE USER ... operation first without a password and then follow up to invoke the new PGConnection.alterUserPassword(...) method.

These new helpers will be available in the next release of the PGJDBC driver.

node.js - pg-password-util

https://www.npmjs.com/package/pg-password-util

On node.js, you can use pg-password-util. This is an NPM module (written by me) that encrypts the password client side and has helpers for updating an existing user's password.

It's designed to work with pg the most popular PostgreSQL driver for node.js. To update a user's password:

import { alterUserPassword } = require('pg-password-util');

// client is a pg.Client
await alterUserPassword(client, {
    username: 'app',
    password: 'my-new-secret-password',
});

Using this helper the plaintext password is never sent over the wire and will not appear in any server logs.

python - psycopg's encrypt_password

https://www.psycopg.org/psycopg3/docs/api/pq.html#psycopg.pq.PGconn.encrypt_password

Python's psycopg driver is a wrapper for libpq (the PostgreSQL C library). It includes a python function wrapping the PQencryptPasswordConn(...) C function mentioned above:

>>> enc = conn.info.encoding
>>> encrypted = conn.pgconn.encrypt_password(password.encode(enc), rolename.encode(enc))
b'SCRAM-SHA-256$4096:...

The return value is a string that can be included in an ALTER USER ... PASSWORD ... statement.

Thanks to Robert Ladyman for suggesting this Python library.

Password Generation

In case it was not obvious, the abcd password above is a terrible password. Real passwords, particularly for service accounts like application database users, should be long and cryptographically random.

On most *nix environments you can generate one using:

tr -d -c a-zA-Z0-9 </dev/urandom | head -c 43

That will generate a random 43-character password. Each character has 62 choices (a-z, A-Z, or 0-9). That's about 5.954 bits of entropy per character (ln(62) / ln2). With 43-characters, that's a little more than 256-bits entropy in total.

Final Thoughts

There are plenty of alternatives to passwords. (I know at least one person who, upon reading this, is already mentally drafting an email to me that everybody should be using Kerberos...) But if you are going to use them, they should not be exposed.

Using the helpers listed above, you could:

Generate a new long random password.
Save the password to a secure secrets store.
Update the database user with the new credentials.

All without persisting the value in plaintext anywhere along the way.

Do you have another library that should be included in this post or want to expand on this further? Let me know!

"ANSI? Schmansi!" at PostgresConf 2018

2018-04-23T00:00:00-04:00

I gave a talk last week at PostgresConf US 2018 titled "ANSI, Schmansi! How I learned to stop worrying and love Postgres-isms". The slides for the talk can be viewed online here:

https://sehrope.github.io/postgres-conf-2018-ansi-schmansi/#/

This talk is about using the full potential of your choice of database with concrete examples of Postgres-specific features. The audience was wonderful and I look forward to more feedback on the topic.

The slides were built with a customized reveal.js template for writing in pug. This allows for most of the example SQL to be housed in separate files (in the sql/ directory) and referenced from the templates. The source for the slides including the example SQL is available here.

Updated 2018-04-26: There's no official video from the conference but a member of the audience recorded a video of the talk and posted it on YouTube. You can watch it here.

Do you think embracing database specific behavior is a great idea? Let me know!

How I Write SQL, Part 1: Naming Conventions

2014-02-16T00:00:00-05:00

Background
Target Audience
Why Naming Conventions Are Important
Naming Conventions
Singular Relations
Key Fields
- Primary Keys
- Foreign Keys
Prefixes and Suffixes (are bad)
Explicit Naming
- Indexes
- Constraints
Final Thoughts

Background

"There are only two hard problems in Computer Science: cache invalidation and naming things."

-- Phil Karlton

In this post I'll be going into the latter. Specifically, I'll describe naming conventions for database objects, why they are so important, and what you should and shouldn't be doing.

Warning! This is a fairly opinionated post and I welcome feedback from people suggesting alternatives.

Target Audience

Our company, JackDB, uses PostgreSQL internally to store persistent state and the naming conventions in this post were written with PostgreSQL in mind. Most of the recommendations should be equally valid for other relational databases such as MySQL, Oracle, or Microsoft SQL Server.

A lot of them will also apply to NoSQL databases, though not everything. For example, the suggestion below to use full english words goes against the recommended approach for naming fields in MongoDB. When in doubt, find a guide for your specific database type.

Why Naming Conventions Are Important

Names Are Long Lived

Data structures are meant to last much longer than application code. Anyone that has worked on a long running system can attest to that.

Well defined data structures and table layouts will outlive any application code. It's not uncommon to see an application completely rewritten without any changes done to its database schema.

Names Are Contracts

Database objects are referenced by their names, thus object names are part of the contract for an object. In a way you can consider your database table and column names to be the API to your data model.

Once they are set, changing them may break dependent applications. This is all the more reason to name things properly before the first use.

Developer Context Switching

Having consistent naming conventions across your data model means that developers will need to spend less time looking up the names of tables, views, and columns. Writing and debugging SQL is easier when you know that person_id must be a foreign key to the id field of the person table.

Naming Conventions

Avoid quotes. If you have to quote an identifier then you should rename it. Quoted identifiers are a serious pain. Writing SQL by hand using quoted identifiers is frustrating and writing dynamic SQL that involves quoted identifiers is even more frustrating.

This also means that you should never include whitespace in identifier names.

Ex: Avoid using names like "FirstName" or "All Employees".

Lowercase. Identifiers should be written entirely in lower case. This includes tables, views, column, and everything else too. Mixed case identifier names means that every usage of the identifier will need to be quoted in double quotes (which we already said are not allowed).

Ex: Use first_name, not "First_Name".

Data types are not names. Database object names, particularly column names, should be a noun describing the field or object. Avoid using words that are just data types such as text or timestamp. The latter is particularly bad as it provides zero context.

Underscores separate words. Object name that are comprised of multiple words should be separated by underscores (ie. snake case).

Ex: Use word_count or team_member_id, not wordcount or wordCount.

Full words, not abbreviations. Object names should be full English words. In general avoid abbreviations, especially if they're just the type that removes vowels. Most SQL databases support at least 30-character names which should be more than enough for a couple English words. PostgreSQL supports up to 63-character for identifiers.

Ex: Use middle_name, not mid_nm.

Use common abbreviations. For a few long words the abbreviation is both more common than the word itself. "Internationalization" and "localization" are the two that come up most often as i18n and l10n respectively. In these cases use the abbreviation.

If you're in doubt, use the full English word. It should be obvious where the abbreviation makes sense.

Avoid reserved words. Avoid using any word that is considered a reserved word in the database that you are using. There aren't that many of them so it's not too much effort to pick a different word. Depending on the context, reserved words may require quoting. This means sometimes you'll write "user" and sometimes just user.

Another benefit of avoiding reserved words is that less-than-intelligent editor syntax highlighting won't erroneously highlight them.

Ex: Avoid using words like user, lock, or table.

Here are the list of reserved words for PostgreSQL, MySQL, Oracle, and MSSQL.

Singular Relations

Tables, views, and other relations that hold data should have singular names, not plural. This means our tables and views would be named team, not teams.

Rather than going into the relational algebra explanation of why this is correct I'll give a few practical reasons.

They're Consistent. It's possible to have a relation that holds a single row. Is it still plural?

They're unambiguous. Using only singular names means you don't need to determine how to pluralize nouns.

Ex: Does a "Person" object go into a "Persons" relation or a "People" one? How about an "Octopus" object? Octopuses? Octopi? Octopodes?

Straightforward 4GL Translation. Singular names allow you to directly translate from 4GL objects to database relations. You may need to remove some underscores and switch to camel case but the name translation will always be straight forward.

Ex: team_member unambigously becomes the class TeamMember in Java or the variable team_member in Python.

Key Fields

Primary Keys

Single column primary key fields should be named id. It's short, simple, and unambiguous. This means that when you're writing SQL you don't have to remember the names of the fields to join on.

CREATE TABLE person (
  id            bigint PRIMARY KEY,
  full_name     text NOT NULL,
  birth_date    date NOT NULL);

Some guides suggest prefixing the table name in the primary key field name, ie. person_id vs id. The extra prefix is redundant. All field names in non-trivial SQL statements (i.e. those with more than one table) should be explicitly qualified and prefixing as a form of namespacing field names is a bad idea.

Foreign Keys

Foreign key fields should be a combination of the name of the referenced table and the name of the referenced fields. For single column foreign keys (by far the most common case) this will be something like foo_id.

CREATE TABLE team_member (
  team_id       bigint NOT NULL REFERENCES team(id),
  person_id     bigint NOT NULL REFERENCES person(id),
  CONSTRAINT team_member_pkey PRIMARY KEY (team_id, person_id));

Prefixes and Suffixes (are bad)

Relation Type Prefixes

Some (older) guidelines suggest naming tables with a TB_ prefix, views with a VW_ prefix, or stored procedures with a SP_ prefix. The rationale being that a programmer reading through some unknown SQL would immediately recognize this and know the object type based on the name. This is a bad idea.

Object names should not include the object type in them. That way you can change it later. A view that is replaced with a table maintains the original contract of a view (ex: you can SELECT from it). A dependent system would not need to be updated after such a change.

I've seen many such systems where at some point a view will become a table. Then you'll end up with code issuing INSERT statements into vw_foobar. There's even a really powerful feature of PostgreSQL that allows you do define DML rules on views (ie. you can INSERT/UPDATE/DELETE from them).

Adding object type prefixes adds extra typing now and extra confusion down the road.

Application Name Prefixes

Another (older) suggestion is to have a common prefix for all your database objects. For example, our app "Foobar" would have tables name Foobar_Users, Foobar_Teams, etc. Again, this is a bad idea.

All modern databases support some form of namespacing. For example, in PostgreSQL you can create schemas to group database objects. If you have multiple applications sharing the same database and want to prevent them from clobbering each other, use schemas instead. That's exactly what they are for!

Most people will not even need them though. It's far more common for a database to be used with a single logical data model than multiple, separate data models. Hence schemas will not be required. If you do need them, it should be fairly obvious.

The exception to this rule is if you are developing a database agnostic code base such as a framework or plugin. Supporting multiple namespacing methods is complicated so many frameworks instead rely on application name prefixing.

However, most people develop applications, not plugins or frameworks, and their applications will reside by themselves in a single type of database. Thus there is no reason to add application name prefixes to all your database objects.

Data Type Suffixes

Some guides (again generally older ones), suggest suffixing your column names with the data type of the field. For example, a text field for a name would be name_tx. There will even be extensive lists to translate from data types to suffixes, text -> tx, date -> dt, etc.

This is a bad idea!

Field data types can change. A date field could become a timestamp, an int could become a bigint or numeric.

Explicit Naming

Some database commands that create database objects do not require you specify a name. An object name will be generated either randomly (ex: fk239nxvknvsdvi) or via a formula (ex: foobar_ix_1). Unless you know exactly how a name will be generated and you are happy with it, you should be explicitly specifying names.

This also includes names generated by ORMs. Many ORMs default to creating indexes and constraints with long gibberish generated names. The couple minutes of time savings in the short run are not worth the head ache in remembering what fkas9dfnksdfnks refers to in the long run.

Indexes

Indexes should be explicitly named and include both the table name and the column name(s) indexed. Including the column names make it much easier to read through SQL explain plans. If an index is named foobar_ix1 then you would need to look up what columns that index covers to understand if it is being used correctly.

CREATE TABLE person (
  id          bigserial PRIMARY KEY,
  email       text NOT NULL,
  first_name  text NOT NULL,
  last_name   text NOT NULL,
  CONSTRAINT person_ck_email_lower_case CHECK (email = LOWER(email)));

CREATE INDEX person_ix_first_name_last_name ON person (first_name, last_name);

Explain plans will now be easy to understand. We can clearly see that the index on first name and last name, ie. person_ix_first_name_last_name, is being used:

=# EXPLAIN SELECT * FROM person WHERE first_name = 'alice' AND last_name = 'smith';

                                          QUERY PLAN                                          
----------------------------------------------------------------------------------------------
 Index Scan using person_ix_first_name_last_name on person  (cost=0.15..8.17 rows=1 width=72)
   Index Cond: ((first_name = 'alice'::text) AND (last_name = 'smith'::text))
(2 rows)

Constraints

Similar to indexes, constraints should given descriptive names. This is especially true for check constraints. It's much easier to diagnose an errant insert if the check constraint that was violated lets you know the cause.

CREATE TABLE team (
  id          bigserial PRIMARY KEY,
  name        text NOT NULL);

CREATE TABLE team_member (
  team_id     bigint REFERENCES team(id),
  person_id   bigint REFERENCES person(id),
  CONSTRAINT team_member_pkey PRIMARY KEY (team_id, person_id));

Notice how PostgreSQL does a good job of giving descriptive names to the foreign key constraints.

=# \d team_member
   Table "public.team_member"
  Column   |  Type  | Modifiers 
-----------+--------+-----------
 team_id   | bigint | not null
 person_id | bigint | not null
Indexes:
    "team_member_pkey" PRIMARY KEY, btree (team_id, person_id)
Foreign-key constraints:
    "team_member_person_id_fkey" FOREIGN KEY (person_id) REFERENCES person(id)
    "team_member_team_id_fkey" FOREIGN KEY (team_id) REFERENCES team(id)

If we try inserting a row that violates one of these constraints we immediately know the cause just based on the constraint name:

=> INSERT INTO team_member(team_id, person_id) VALUES (1234, 5678);
ERROR:  insert or update on table "team_member" violates foreign key constraint "team_member_team_id_fkey"
DETAIL:  Key (team_id)=(1234) is not present in table "team".

Similarly, if we try inserting an email address that is not lower case into the person table created above, we'll get a constraint violation error that tells us exactly what is wrong:

-- This insert will work:
=> INSERT INTO person (email, first_name, last_name) VALUES ('alice@example.com', 'Alice', 'Anderson');
INSERT 0 1

-- This insert will not work:
=> INSERT INTO person (email, first_name, last_name) VALUES ('bob@EXAMPLE.com', 'Bob', 'Barker');
ERROR:  new row for relation "person" violates check constraint "person_ck_email_lower_case"
DETAIL:  Failing row contains (2, bob@EXAMPLE.com, Bob, Barker).

Final Thoughts

If you're starting a new project then I suggest you follow the conventions outlined here. If you're working on an existing project then you need to be a bit more careful with any new objects you create.

The only thing worse than bad naming conventions is multiple naming conventions. If your existing project already has a standard approach to naming its database objects then keep using it.

Do you have something to add to this list, a way to improve some of these guidelines, or just think some of these are terrible? Let me know!

AWS Tips, Tricks, and Techniques

2014-01-29T00:00:00-05:00

Background

AWS is one of the most popular cloud computing platforms. It provides everything from object storage (S3), elastically provisioned servers (EC2), databases as a service (RDS), payment processing (DevPay), virtualized networking (VPC and AWS Direct Connect), content delivery networks (CDN), monitoring (CloudWatch), queueing (SQS), and a whole lot more.

In this post I'll be going over some tips, tricks, and general advice for getting started with Amazon Web Services (AWS). The majority of these are lessons we've learned in deploying and running our cloud SaaS product, JackDB, which runs entirely on AWS.

Billing

AWS billing is invoiced at the end of the month and AWS services are generally provided on a "per use" basis. For example EC2 servers are quoted in $/hour. If you spin up a server for 6 hours then turn it off you'll only be billed for those 6 hours.

Unfortunately, AWS does not provide a way to cap your monthly expenses. If you accidentally spin up too many servers and forget to turn them off, then you could get a bit shock at the end of they month. Similarly, AWS charges you for total outbound bandwidth used. If you have a spike in activity to a site hosted on AWS or just excessive usage of S3, you could end up with a sizable bill.

AWS does allow you to set up billing alerts. Amazon CloudWatch allows you to use your projected monthly bill as a metric for alerts. You can have a notification sent to you when it exceeds a preset dollar amount.

Do this immediately!

Seriously just go add this immediately. Even better, add a couple of these at a variety of dollar figures. A good starting set is: $1/mo, $10/mo, $50/mo, $100/mo, $250/mo, $500/mo, and $1,000/mo

If you ever end up with a run away server or accidentally provision 10 servers instead of 1 (which is surprisingly easy when scripting automated deployments...), then you'll be happy you set up these billing alerts.

Our company, JackDB, is entirely hosted on AWS and our bills are fairly consistent month to month. Once that monthly bill stabilized we setup billing alerts, as described above, at the expected value as well as 1/3 and 2/3 of it. This means that on approximately the 10th and 20th of each month we get a notification that our bill is at 1/3 or 2/3 of our expected monthly spend.

If either of those alerts is triggered sooner than that, it'd be a sign that monthly is on the rise.

Security

Security is a big deal, especially so in the cloud. Countless articles have been written about it and the advice here is just a couple quick points. At a later date I'd like to write a more detailed write up.

Multi-factor Authentication

Mutli-factor authentication, also known as two-factor authentication or 2FA, is an enhanced approach to authentication that requires combining multiple, separate, factors of authentication. Most commonly these are something you know (ex: a password) and something you have (ex: a hardware token or virtual token generator on your phone).

If one of these gets compromised (ex: someone steals your password), they would still need one of the other factors to login. A single factor alone isn't enough.

AWS supports multi-factor authentication using standard TOTP pin codes. It supports both free software pins (ex: Google Authenticator on your smart phone) and hardware tokens ($12.99 as of Jan, 2014).

Do this immediately!

There is no reason not to have this enabled and I recommend immediately enabling it. In fact, you should enable 2FA on every service you use that supports it. If you're using Google Apps or even just regular Gmail, you should enable it there as well.

SSH

It's a good idea to use unique SSH keys for unrelated projects. It makes it much easier to deprovision them later on. You should generate a new pair of SSH keys (with a long passphrase) to use with your new servers. If you have multiple people sharing access to the same servers, each person should have their own unique SSH keys.

Rather than specifying your key files on the command line, you should add them to your ~/.ssh/config file so they'll be automatically used. q

Host ec2-10-20-30-40.compute-1.amazonaws.com
  User ubuntu
  IdentityFile "~/.ssh/my-ec2-private-key"

By default SSH will send all your available SSH keys in your keyring or listed in your config file to a remote server. If you have a lot of SSH keys then you may get an error from a remote server when you try to SSH to it:

Too many authentication failures for username

From the remote server's perspective each is considered a connection attempt. If you have too many SSH keys for it to try then it may not get to the correct one. To force it to only send the SSH key specific to the server you are connecting to as listed in your config file, add the following to the top of your ~/.ssh/config:

Host *
  IdentitiesOnly yes

This will force you to explicitly list the SSH key to use for all remote servers. If you want to restrict this to just a subset of them, you can replace the "*" in the Host section with a wildcard matching the DNS name of the servers. For example *.example.com.

VPC

Amazon Virtual Private Coud (VPC) is a networking feature of EC2 that allows you to define a private network for a group of servers. Using it greatly simplifies fencing off components of your infrastructure and minimizing the externally facing pieces.

The basic idea is to separate your infrastructure into two halves, a public half and a private half. The external endpoints for whatever you are creating goes in the public half. For a web application this would be your web server or load balancer.

Services that are only consumed internally such as databases or caching servers belong in the private half. Components in the private half are not directly accessible from the public internet.

This is a form of the principle of least privilege and it's a good idea to implement it. If your server infrastructure involves more than one server, then you probably should be using a VPC.

Bastion Host

To access internal components in the private half your VPC you'll need a bastion host. This is a dedicated server that will act as an SSH proxy to connect to your other internal components. It sits in the public half of your VPC.

Using a bastion host with a VPC greatly simplifies network security when working on AWS by significantly minimizing the number of external firewall rules you need to manage. Here's how to set one up:

Spin up a new server in the public half of your VPC
Create a security group Bastion SG and assign it to the new server
Edit the security groups for your private half servers to allow inbound access on port 22 (SSH) from Bastion SG
Add your whitelisted IPs to the inbound ACL for Bastion SG (see the next section)

An SSH proxy server doesn't use that much CPU and practically zero disk. A m1.micro instance (the cheapest one that AWS offers) is more than enough for this. As of Jan 2014, at on-demand rates this comes out to ~$15/mo. With reserved instances you can bring this down to about $7/mo.

The added cost is nothing compared to the simplicity and security it adds to your overall environment.

Firewall Whitelists

Any server with port 22 (SSH) open to the public internet will get a lot of hacking attempts. Within 24 hours of turning on a such a server you should see a lot of entries in your SSH logs of bots trying to brute force log in. If you only allow SSH key based authentication this will be a pointless exercise but it's still annoying to see all the entries in the log (ie. it's extra noise).

One way to avoid this is to whitelist the IP addresses that can connect to your server. If you have a static IP address at your office or if it's "mostly static" (ex: most dynamic IPs for cable modems and DSL don't change very often), then you can set up the firewall rules for your servers to only allow inbound SSH access from those IPs. Other IP addresses will not even be able to tell there is an SSH server running. Port scanning for an SSH server will fail as the initial TCP socket will never get established.

Normally this would be a pain to manage on multiple servers but by using a bastion host this only needs to be done in one place. Later on if your IP address changes or you need to connect to your server from a new location (ex: on the road at a hotel), then just add your current IP address to the whitelist. When you're done, simply remove it from the list.

Email

Amazon Simple Email Service (SES) is Amazon's send-only email service for AWS. You can use it to send out email from your application or command line scripts. It includes both an AWS specific programmatic API as well as a standard STMP interface. Both cost the same (it's pay per use) but for portability purposes I recommend using the SMTP interface. This allows you to change your email provider down the road.

If you're only sending a handful emails then SES can be used free of charge. The first 2,000 emails per day are free when you send them from an EC2 server. For a lot of people this should be more than enough. After the first 2,000 per day it's $.10 per 1,000 emails plus outbound bandwidth costs.

Verification

When you first set it up you'll need to verify ownership for the email addresses that you'll be sending from. For example if you want to send email from hello@example.com then you'll need to prove you actually control example.com. This is done by either verifying that you can receive email at that address. You can also verify an entire domain by setting up TXT records to verify domain ownership.

You cannot send email until this verification is complete and it can take a little while for it to propagate through the system. Additionally, to prevent spammers from using SES for nefarious purposes, Amazon restricts your initial usage of SES. Your sending limit is gradually increased. If you plan on sending email from your application you should set up SES immediately so that it's available when you need it.

DKIM and SPF

To ensure that your emails are actually received by your recipients and not rejected by their spam filters you should set up both DKIM and SPF for your domain. Each is a way for recipients to verify that Amazon SES is a legitimate sender of email for your domain.

You can read more about setting up SPF on SES here.

You can read more about setting up DKIM on SES here.

Also, if you haven't already, you should set up DKIM and SPF for your domain's email servers as well. If you're using Google Apps for email hosting more details are available here.

Testing via Port 25

Once you have it set up, a simple way to test out both DKIM and SPF is using the email verification service provided by Port 25. Simply send them an email and a short while later they'll respond back with a report saying whether SPF and DKIM are properly configured. They'll also indicate whether their spam filters would flag your message as junk mail.

Note that you can also use Port 25 to verify your personal email address as well, not just Amazon SES. Just manually send an email to Port 25 and wait for the response.

EC2 (on the cheap)

Amazon EC2 allows you to spin up servers on demand and you only pay for what you usage, billed hourly. This means it's particularly catered towards usage patterns that involve scaling up to a large number of servers for a short period of time.

The flip side of the fine grained billing of EC2 is that the on-demand price of the servers is more expensive than other cloud providers. Here a couple tips to lower your costs.

Reserved Instances

If you are running a server that is always online you should look into reserved instances. With reserved instances you pay an upfront fee to in exchange for greatly reduced hourly rates. Amazon offers three levels of reserved instances, Light (the cheapest), Medium, and Heavy (most expensive), and each is offered in 1-year (cheaper) or 3-year blocks (more expensive).

Each costs progressively more, but also progressively reduces the hourly cost for running a server. For example a standard m1.small instance costs $.06/hour on-demand. If you buy a 3-year light reserved instance for $96 the hourly price drops to $.027/hour. Factoring in the up front cost and assuming the server runs 24-hours a day, it would take about 4 months to break even with the reserved instance vs paying on-demand rates.

Billing for Heavy instances is a bit different than Light or Medium as you'll be billed for the underlying instances regardless of whether it's actually running. This means that if you buy a 3-year heavy reserved instance you're agreeing to pay for running that instance 24-hours a day for the next three years.

The savings for reserved instances is anywhere from 25% to 65% over on-demand pricing. The break even point is anywhere from 3-months to a year. Once your AWS usage has stabilized it's well worth the time to investigate the cost savings of buying reserved instances.

If you're unsure of whether you'll be running the instance types a year or two from now I suggest sticking to Light or Medium instances. The savings differential is only another 10-15% but they allow you more cheaply change your infrastructure down the road.

Reserved Instance Marketplace

There's also a reserved instance marketplace where you can buy reserved instances from third parties or sell those that you no longer need. Since reserved instances only impact billing there's no difference in functionality in buying them via the third party marketplace. In fact, it's all part of the same UI in the admin console.

The only real difference with third party reserved instances is the duration of the reservation can be just about anything. This can be very useful if you anticipate a usage term besides 1-year or 3-years. Just make sure to check the actual price as there usually a couple listed that have wildly inflated prices compared to the Amazon offered ones.

Spot Instances

Spot instances allow you to bid on excess EC2 capacity. As long as your bid price is above the current spot price, your instance will continue to run and you'll pay the lower of the two per hour. If the spot price increases beyond your bid, your instance may be terminated. This termination could happen at anytime.

As they can be terminated at any time, spot instances work best when application state and results are stored outside the instance itself. Idempotent operations and reproducible or parallelizable work is usually a great candidate for spot instances. One of the most popular use cases for it is for running continuous integration (CI) servers. If the CI server crashes and restarts an hour later, it's not that big of a deal. The only result we really care about is if the final application build/test was successful.

More complicated setups are possible with spot instances as well. If you fully automate their provisioning (ie. automate security groups, user data, startup scripts), you can even have them automatically register themselves via Route53 DNS and then join an existing load balancer. In a later article I'll be writing about how to build such a setup that cheaply scales out a stateless web service via a phalanx of spot instances, all the while being fault tolerant to their random termination.

S3

S3 is Amazon's object store. It allows you to store arbitrary objects (up to 5TB in size) and access them over HTTP or HTTPS. It is said to be designed to provide 99.999999999% durability and 99.99% availability. For what it offers, S3 is really cheap. Amazon also periodically drops the prices for it as well. Most recently a week ago.

The S3 API allows you to create signed URLs that provide fine grained access to S3 resources with custom expirations. For example you can have an object stored in S3 that is not publicly readable to be temporarily accessible via a signed URL. This is a great way to provide user's access to objects stored in private S3 buckets from webapps.

GPG

If you're using S3 to store sensitive data (ex: database backups) then you should encrypt the data before you upload it to S3. That way only encrypted data is stored on Amazon's servers.

The easiest way to do this is using GPG. Generate a GPG key pair for your backup server on your local machine and add the public key of the key pair to the server. Then use that public key to encrypt any data before uploading to S3.

# Set the bucket/path on S3 where we will put the backup:
$ S3_PATH="s3://my-s3-bucket/path/to/backup/backup-$(date +%Y%M%d)"

# Temp file for encryption:
$ GPG_TEMP_FILE=$(mktemp)

# Encrypt it with GPG:
$ gpg --recipient aws.backup@example.com --output "$GPG_TEMP_FILE" --encrypt mydata.foo

# Upload it via s3cmd:
$ s3cmd put "$GPG_TEMP_FILE" "$S3_PATH"

# Clean up temp file:
$ rm "$GPG_TEMP_FILE"

The only downside to encrypting data prior to storage on S3 is that you will need to decrypt it to read it. You can't provide the S3 URL to someone else to download the data (ex: a direct link to a user's content that you're storing on their behalf) as they will will not be able to decrypt it. For backups this is the right approach as only you (or your company, your team, ...) should be able to read your data.

The S3 CLI tool s3cmd includes an option to specify a GPG key. If set, then it will automatically encrypt at objects you PUT to S3. It can be a convenient option as you only need to set it one place (the s3cmd config file). I prefer explicitly adding the GPG steps to my backup scripts though.

Encryption At Rest

S3 also supports what is referred to as Server Side Encryption . With this option enabled, Amazon will encrypt your data before persisting it to disk, and will transparently decrypt it prior to serving it back to a valid S3 request. The documentation for it describes how they keep the decryption keys separate from the S3 API servers and request them on demand.

Since Amazon can still read your data I don't consider this to be that useful of a feature. If you want to ensure that nobody else can read your data then you need to do the encryption. If you trust a third party to do it, then by definition that third party is able to read your unencrypted data.

Still though, it doesn't hurt to enable it either. Apparently there is no real performance penalty for enabling it.

Object Expiration

Once you start using S3 for backups you'll notice that your S3 bill will grow fairly linearly (or faster!). By default, S3 persists objects forever so you'll need to take some extra steps to clean things up.

The obvious solution is to delete objects as they get old. If you're using S3 for automated backups then you can have your backup script delete older entries. With additional logic you can have your backup scripts keep various ages of backups (ex: monthly for a year, weekly for a month, daily for a week).

A simpler, albeit coarser, approach is to use S3 object expirations. It allows you to define a maximum age for S3 objects. Once that age is reached, the object will automatically be deleted with no user interaction. You could set a 6-month expiration on your S3 backup bucket and it will automatically delete older entries than that.

Warning: If you use S3 object expiration make sure that your backups are actually working. It will delete your old objects regardless of whether your latest backups are actually valid. Make sure to test your backups regularly!

Object expiration is also a simple way to delete all the objects in an S3 bucket. From the AWS S3 console simply set the expiration for the entire bucket to be 1-day. The objects will then be automatically deleted after a day. S3 expiration is asynchronous so you may still see the objects in S3 for a short while after 24-hours but you will not be billed for them.

Glacier

Glacier is Amazon's long term, persistent storage service. It's about an order of magnitude cheaper than S3 but with much slower access times (on the order of multiple hours).

Rather than deleting old S3 objects you can configure S3 to automatically expire them to Glacier. The storage cost will be about 10x less and if you really need the data (ex: you accidentally destroyed all your S3 backups) you can slowly restore them from Glacier.

Final Thoughts

This post ended up longer than originally planned, but it's still a very incomplete list. There are plenty of other tips, tricks, and techniques to use with AWS, this is just a (hopefully helpful) start.

Do you have something to add to this list, a better way of solving some of these problems, or just want to be notified when I have a new post? Let me know!

My blog's tech stack: Pelican powered, Dokku deployed

2014-01-23T00:00:00-05:00

Background
Requirements
- Cheap
- Easy
- New
- Fun
Static Site
- Generators
- Pelican
Setup
Final Thoughts

Background

This post goes into the technical components of setting up this blog and some of the merits of the tech stack choices.

For the impatient, the final choice of stack is a Pelican powered static site, styled with a modified Octopress theme, deployed via Dokku, running nginx in a Docker container, and it's all deployed to DigitalOcean droplet.

To put it another way: it's a riddle, wrapped in a mystery, inside an enigma.

Requirements

When setting up a new blog one of the most important (and fun) decisions you get to make is the tech stack that will run it. Besides the topic and posts themselves, this is probably the most important decision. Additionally, the more technologically inclined you are, the more time you'll spend on this decision.

In my case, I'd use a number of platforms in the past and had reduced my core requirements list to the following (in order of importance):

Cheap
Easy
New
Fun

Cheap

In 2014 there are many options for cheap hosting so cost was not much of a concern.

For static content there are a number of free hosting options such as Tumblr or GitHub pages. You can also use services like Amazon S3 to host a website for next to nothing.

For example our company's website, https://www.jackdb.com/, is a static site hosted on Amazon S3. The bill for it is literally pennies a month and, since it's on S3, we never worry about scaling it up to handle large influxes of traffic.

Even sites with dynamic content can be hosted for free or relatively cheaply. Heroku gives you a free web dyno per application. You use it to either run a dynamic site (ex: Wordpress or Ghost) or a static site (ex: nginx serving a pregenerated site).

Easy

Easy in this context means it should be easy to preview an article while writing and the final deployment step should be automated. The ideal workflow for writing will look something like this:

Write something
View what it'll look like in the browser
Not done? Goto step 1
Publish to the production site

Saavy readers will recognize this as the writing equivalent of a REPL. Since a big part of writing is minimizing distraction (so you can actually focus on writing), it's important to have an efficient way to "test" your posts (i.e. the step 2 to 3 flow).,

Publishing should be as simple as possible, preferably from the command line. This usually means it should be git push style deploy.

Note that being "easy" does not refer to the initial setup of the platform. On the contrary, the newer the platform the more likely there will be complications with the initial setup. If they're not show stoppers then that would probably increase the "fun" factor though.

New

By "new" I'm referring to technology choices that are new to not just me, but the world as a whole. Like any good technologist I'd like to use this as an opportunity to experiement with new technologies.

There's a high correlation between new and fun.

Fun

The "fun" factor of setting up a new website is derived from the new (to you) technology you get to experiment with. The more twists, turns, and hacks to set it up, the more fun you're going to have. A decent gauge of whether something is going to be fun is whether you'd want to listen to someone else describe the set up process.

On the low end of the fun scale (arguably negative fun...) is using a hosted service like Tumblr. There is no real set up work involved (just sign up and optionally pick a theme) and nothing new to experiment with.

The opposite extreme would be writing your own blogging software. It would be a very fun exercise and would make for a good write up on its own, but you probably would not get much writing done until it's finished.

Static Site

I decided early on that I wanted the site to be entirely static. A static site increases your hosting options and will generally scale very cheaply and easily. Properly deployed (i.e. ngninx with the right cache headers), a static site on a single server should handle any reasonable amount of traffic. Deciding on having the site be static also helps with the paradox of choice in picking a tech stack.

The biggest limitation with a static site is that all the content that you serve is generated in advance. This means you can't have any dynamicly generated content and any user interaction with your site will be handled by external services.

For example, if you want to add comments to a static website you could use a service like Disqus. If you'd like to have a sign up form to collect email addresses from your readers, you can use a service like Mailchimp or Wufoo.

Generators

The end result of a static website is a collection of HTML, CSS, and other asset files that you'd deploy to a web server. Rather than generate them by hand (i.e. copy paste each page...) most people use a static site generator. These are pieces of software that automate the process of generating the HTML pages of your site.

Some popular features for static site generators are:

Allow writing posts in a light weight syntax such as Markdown rather than straight HTML
Automatically appying common CSS styling and theming to all your posts
Generating index and listings pages
Grouping pages based on post tags or content

See here for a great list of alternatives. There are a lots of them.

Pelican

The static site generator I decided to use is Pelican. Here are some quick highlights of it:

Written in Python
Under active development
Lots of themes available
Many deployment options (the final result is a static site)
Easy to install

I don't write a large amount of Python code but it is a language I like a lot. The syntax is clean, it has a wealth of libraries, and it runs on just about every platform. I didn't have to modify the source for Pelican in setting up my blog but, if I did, I know it'll be pretty easy.

Although I didn't particularly like Octopress, I do like the standard Octopress theme. Thankfully, there is a Pelican theme for it! I used it as a base and added/updated/fixed a few things. You can find my fork here.

Setup

Although installing Pelican was easy enough, getting a sample site up took a bit of work. All the building blocks are there but the documentation and initial setup could use a bit of work.

Pelican Quickstart

Pelican includes an executable, pelican-quickstart, that automates the creation of a new site. It asks you a couple basic questions (your name, site name, URL, etc) and creates an site directory. It even includes a development script that watches your file system for changes and automatically regenerates the static site (REPL solved!).

Unfortunately, the site it generates is really empty. A lot of the standard customizations that you'd expect to do to a new site are missing:

No sample post (where do I write?)
No favicon defined (where does it go?)
No sample images (where am I suppoed to add them?)
No sample extras (where do I put misc files I want to add like a PGP key?)
No markdown code highlighting (how do I enable this?)

Barebones Pelican Blog

It's perfectly understandable why the base site that is generated doesn't include any of these. It's meant to be as lean as possible and added atop.

Still though, I prefer a working example I can modify. When working with a new piece of technology it's always easier to tweak a functional existing system and comment out blocks you don't need then add things you do.

To simplify this for the next person who'd like to use Pelican, I created a bare bones example of a Pelican powered blog. You can find it here: https://github.com/sehrope/pelican-dokku-bare.

Some highlights of it are:

Includes a customization list (i.e. list of places to edit to make it yours)
Includes a sample post
Includes a sample image (and how to link to it in a post)
Includes a sample PGP key (to show how to include extra files)
Enables markdown code highlighting
Custom 403, 404, and 500 error page via git
Defaults to my tweaked Octopress theme
Deploys out of the box to a Paas (i.e. Heroku or Dokku) via git

To use it just clone the repo and start it up:

# Clone it locally:
$ git clone https://github.com/sehrope/pelican-dokku-bare my-blog

# Enter the cloned directory:
$ cd my-blog

# Start up the development server:
$ ./develop_server.sh start

Now if you add or edit a file in the content/posts directory it will automatically show up. To further customize it follow the instructions in the README.

Docker, Dokku, and DigitalOcean

The final (and most fun) part of the setup was the install for Docker and Dokku on DigitalOcean. Docker and Dokku definitely fall into both the "new" and "fun" categories of technology. It was so fun I wrote about setting it all up and how to improve it further.

In reality the Dokku server set up was the first thing I did — the blog itself was started because I wanted to write about the setup.

Once Dokku is configured, this blog is just another application deployed to my DigitalOcean droplet. There is no special setup or configuration for it. I just added it as a git remote and git push to it whenever I'd like to publish new content.

All it takes is:

# Add the remote Dokku production site:
$ git remote add production dokku@launchbylunch.com:launchbylunch.com

Then I can publish the blog via:

$ git push production master

Publishing takes about a minute as each time it rebuilds the entire application from scratch. The Pelican site generation takes barely a second and most of the deploy time is spent building the dependencies (ex: the buildpack compiles nginx from scratch). It's all fully automated and reproducible though so you don't really think about it.

SSL

The only addtional configuration I did for this site after deploying it to my DigitalOcean droplet was to add SSL. Dokku has support for it out of the box so it was really just a matter of creating the server's private key, getting it signed, and uploading it.

First generate a 4096 bit key. You'll be prompted for a passphrase (don't forget it!).

$ openssl genrsa -des3 -out server.key.secure 4096
Generating RSA private key, 4096 bit long modulus
[... truncated ...]
Enter pass phrase for server.key.secure:
Verifying - Enter pass phrase for server.key.secure:

Then create a password-less version of the private key. This will be the file we upload to our server. It doesn't have a password so that our web server can be restarted without manual intervention.

$ openssl rsa -in server.key.secure -out server.key
Enter pass phrase for server.key.secure:
writing RSA key

Finally, create the certificate signing request. Most of the fields are arbitrary. The important one is the "Common Name" which should match the hostname of your server:

$ openssl req -new -key server.key.secure -out server.csr
Enter pass phrase for server.key.secure:
You are about to be asked to enter information that will be incorporated
into your certificate request.
[... truncated ...]
Common Name (e.g. server FQDN or YOUR name) []:launchbylunch.com
[... truncated ...]

Now you should have three files:

__server.key.secure __ - the private key
server.key - the private key with the password removed
server.csr - the certificate signing request for you key

Next, you'll need to have your server.csr signed by a certificate authority. I use Namecheap as my registrar and they offer $1.99 SSL certifcates (for one year) when you buy a domain. I've got a bunch of them accumulated so I used one of those.

In 10-15 minutes you'll receive a confirmation email to prove domain ownership. Once that is complete you'll receive a reply from the certificate authority with the signed certificate. If they include an intermediate certificate (ex: the Namecheap Positive SSL I describe above includes two) then concatenate them into a single server.crt.

Finaly upload server.key and server.crt to your server and place the two files in ~dokku/<app>/ssl and redeploy the app. In my case that's ~dokku/launchbylunch.com/ssl.

Upon startup Dokku will detect the SSL certificate and configure it's nginx router to redirect traffic from port 80 (HTTP) to 443 (HTTPS). It also does a good job of setting up nginx with PFS. The only thing it doesn't include out of the box is HSTS but we'll add that later.

Final Thoughts

The end result of all this is that I have an easy to deploy to blog running for $10/mo. In reality I could host it for free (it's static content after all) but I like the control of running the server myself (and it's way more fun). Plus, as it's just one of the many Dokku deployed apps on my server, the cost itself is shared, and I have an excuse to keep the server running.

Pelican itself is really pleasant to use. Besides a couple hiccups in the initial setup I haven't had any issues with it and would recommend it to anyone else setting up a blog.

Did you try this out or do you have a more cool/fun/interesting setup for your site? Tell me about it.

Encrypting Docker containers on a Virtual Server

2014-01-13T00:00:00-05:00

Background
Why Encrypt?
Limitations
Setup
Destroying Everything
Final Thoughts

Background

In my last post I went over setting up Dokku on a DigitalOcean droplet. In this post we'll improve that setup a bit by add some encrypted swap space. We'll also be encrypting all Dokku files, Docker containers, and the application log files for our deployed Dokku apps.

Why Encrypt?

Encryption is used to prevent unauthorized people from reading your data. When done at the file system level, encryption has the additional benefit that it makes it much easier to dispose of the data on a disk.

An import part of disposing of any server (or more generally any hard drive) is wiping the disks. With a cloud server, you are not in custody of the physical disks and securely wiping them becomes even more important. If you're not careful about it then bad things can happen.

To wipe an unencrypted disk you would need to over write the entire disk with zeroes (or random gibberish from /dev/urandom). Otherwise any data that was previously there could be retrieved. The larger the disk, the longer this will take. For example a 1 TB disk with a sequential write speed of 75 MB/s would take almost 4 hours to zero out.

On the other hand, with an encrypted disk you can simply destroy the encryption keys and nobody will be able to read the original data again. Since the encryption keys are relatively small (ex: the header and key slots for LUKS total only a couple of MB), zeroing them out is very quick.

Limitations

The steps in this guide will setup an encrypted block device. The encryption key for the block device will be stored in plaintext along side it. This is being done for convenience and will allow the virtual server to reboot without manual intervention (i.e. we don't have to enter the pass phrase at boot time).

This is not secure!

Just to clarify further ... This is not really secure! Having the key stored in plain text alongside the encrypted block device is kind of like having a lock on your door and the key on the door step. If someone has access to your virtual server then they will be able to get the decryption keys. This includes any backup snapshots that archive the virtual server's entire disk.

Still though, this is better than nothing and will allow you to more easily destroy your virtual server's disk when you're done using it. At the end of this post I'll go over some additional ideas to improve this configuration further.

Setup

A standard DigitalOcean droplet comes with a single disk allocated with the full space of the droplet. In our case it's approximately 30 GB:

root@launchbylunch:~# df -h
Filesystem           Size  Used Avail Use% Mounted on
/dev/vda              30G  6.1G   22G  22% /
none                 4.0K     0  4.0K   0% /sys/fs/cgroup
udev                 494M  4.0K  494M   1% /dev
tmpfs                100M  248K  100M   1% /run
none                 5.0M     0  5.0M   0% /run/lock
none                 498M  7.0M  491M   2% /run/shm
none                 100M     0  100M   0% /run/user

Since we don't have a separate hard drive (it's all one disk), we'll be creating a file on the existing non-encrypted filesystem and using that as a block device (via a loop device).

Install Packages

There's only one package to install, cryptsetup. Install it via apt-get:

$ apt-get -y install cryptsetup

Create Block Device Files

Throughout this post I'll be using $CRYPTFS_ROOT as the path on the unecrypted filesystem. If you'd like to have the encrypted block device and key stored elsewhere, simply modify the line that sets it's value.

Create a directory at the root of the filesystem to hold our encrypted block device.

# Location on the root filesystem where we'll store everything:
$ CRYPTFS_ROOT=/cryptfs

# Make the directory to store our encrypted block device:
$ mkdir -p $CRYPTFS_ROOT

Next, we'll create the encryption key for our block device. The key will be randomly generated from /dev/urandom. For most purposes this should be fine but for a more secure setup this key should generated on a non-virtual server (ex: your desktop) and uploaded to the virtual server.

# Create a random encryption key:
$ dd if=/dev/urandom of=$CRYPTFS_ROOT/key bs=4K count=1

Next, we'll create a 2 GB swap file. We'll be initializing it with zeroes so that it gets fully allocated. This will take up to 10-20 seconds as it will need to write everything to disk.

# Create and allocate a 2 GB swap file:
$ dd if=/dev/zero of=$CRYPTFS_ROOT/swap bs=1M count=2048

Next, we'll create our encrypted block device. Since we don't really care about the existing data on the disk we'll create a sparse file using the truncate command. Alternatively, if we wanted to overwrite any existing data on disk we could allocate it similar to how the swap file was allocated (though this would take substantially longer).

# Create a sparse 20 GB file:
$ truncate -s 20G $CRYPTFS_ROOT/disk

Finally we'll change the permissions on all these files so only root can access them:

$ chmod -R 700 "$CRYPTFS_ROOT"

Setup Swap

Now we'll add an entry to your /etc/crypttab for a swap file. We'll also add an entry to /etc/fstab so that the swap is automatically enabled at each boot.

The encryption key for this will be randomly chosen at each boot from /dev/urandom and never saved. This means that if the virtual machine is powered off no application data that was written to swap will be recoverable (that's a good thing!).

# Add a line to /etc/crypttab for our swap file:
$ echo "swap $CRYPTFS_ROOT/swap /dev/urandom swap" >> /etc/crypttab

# Add swap entry to /etc/fstab:
$ echo "/dev/mapper/swap none swap defaults 0 0" >> /etc/fstab

Encrypted Block Device Setup

Next up we'll create a LUKS volume in our 20 GB file. To do so we'll first need to mount it using a loop device.

# Pick an unused loop back device and save it to a variable:
$ LOOP_DEVICE=$(losetup -f)

# Mount our block device file to it:
$ losetup $LOOP_DEVICE $CRYPTFS_ROOT/disk

Now we'll use cryptsetup to format it as a LUKS volume. We'll be using the default settings for LUKS.

# Create a LUKS volume with the default options using our key file
$ cryptsetup --batch-mode --key-file=$CRYPTFS_ROOT/key luksFormat $LOOP_DEVICE

# Open it up so it appeares in /dev/mapper
$ cryptsetup luksOpen --key-file=$CRYPTFS_ROOT/key $LOOP_DEVICE cryptfs

The encrypted block device should now show up as /dev/mapper/cryptfs. We can check the cipher details for it by running cryptsetup status. It should look like this:

$ cryptsetup status cryptfs
/dev/mapper/cryptfs is active and is in use.
  type:    LUKS1
  cipher:  aes-cbc-essiv:sha256
  keysize: 256 bits
  device:  /dev/loop0
  loop:    /cryptfs/disk
  offset:  4096 sectors
  size:    41938944 sectors
  mode:    read/write

Now we'll create a filesystem on the encrypted block device. We're creating an ext4 filesystem:

$ mkfs.ext4 /dev/mapper/cryptfs

Finally lets create a mount point for it and add it to /etc/crypttab and /etc/fstab so that it gets automatically mounted at boot time.

# Create a mount point for the encrypted filesystem:
$ mkdir -p /mnt/cryptfs

# Add to /etc/crypttab using the fixed key at $CRYPTFS_ROOT/key
$ echo "cryptfs  $CRYPTFS_ROOT/disk  $CRYPTFS_ROOT/key  luks" >> /etc/crypttab

# Add it to /etc/fstab:
$ echo "/dev/mapper/cryptfs  /mnt/cryptfs  ext4  defaults  0  1" >> /etc/fstab

Now that the setup is complete let's reboot the server once and test it out. If everything was properly set up, then after the reboot the encrypted filesystem should be mounted to /mnt/cryptfs and we should have 2 GB of swap:

# Verify that the encrypted block device was mounted:
$ mount | grep cryptfs
/dev/mapper/cryptfs on /mnt/cryptfs type ext4 (rw)

$ free -m
             total       used       free     shared    buffers     cached
Mem:           995        514        480          0         86        182
-/+ buffers/cache:        245        749
Swap:         2047          0       2047

Docker and Dokku Setup

At this point we have a mount point at /mnt/cryptfs that will transparently encrypt and decrypt any data stored there. You could symlink other parts of your filesystem to subdirectories there to have them encrypted as well.

# Make a directory in the encrypted filesystem
$ mkdir -p /mnt/cryptfs/foo

# Create a symlink pointing to it for /var/lib/foo:
$ ln -s /mnt/cryptfs/foo /var/lib/foo

Unfortunately, this does not currently work with volume mounts for Docker containers(see this issue). A work around though is to use bind mounts. Bind mounts allow you to mount one part of a file system to another location. Conceptually they're very similar to symlinks.

Bind Mounts

We'll now add bind mounts to our /etc/fstab for all the directories that we'd like to be stored on our encrypted filesystem. This will include our all our Docker and Dokku related files, all /home directories, and the eventual location of nginx and our Dokku application log files.

WARNING: This guide assumes that you've started with an empty server. Any existing data in /home will be inaccessible after adding these bind mounts. A new empty home directory will take it's place. Also, this must be done before you install Dokku or Docker as both of their installation directories will also be emptied.

# Loop through the paths that we'd like to be encrypted:
for DIR_NAME in home var/lib/docker var/lib/dokku var/log/dokku var/log/nginx
do
  echo "Adding bind mount for DIR_NAME=$DIR_NAME"
  # Create the path on the encrypted filesystem (if it doesn't exist)
  mkdir -p "/mnt/cryptfs/${DIR_NAME}"

  # Create the path on the actual filesystem (if it doesn't exist)
  mkdir -p "/$DIR_NAME"

  # Add a bind mount entry to /etc/fstab for it:
  echo "/mnt/cryptfs/${DIR_NAME} /$DIR_NAME none bind 0 0" >> /etc/fstab
done

Updated @ 2014/Mar/07: A previous version of this post incorrectly had the directories for the bind mount backwards. Thanks to John (https://github.com/ubergarm) for pointing this out!

Now let's reboot one more time. After rebooting, verify that everything was mounted properly.

$ mount
/dev/vda on / type ext4 (rw,errors=remount-ro)
[ ... truncated ...]
/dev/mapper/cryptfs on /mnt/cryptfs type ext4 (rw)
/mnt/cryptfs/home on /home type none (rw,bind)
/mnt/cryptfs/var/lib/docker on /var/lib/docker type none (rw,bind)
/mnt/cryptfs/var/lib/dokku on /var/lib/dokku type none (rw,bind)
/mnt/cryptfs/var/log/dokku on /var/log/dokku type none (rw,bind)
/mnt/cryptfs/var/log/nginx on /var/log/nginx type none (rw,bind)

Installation

Now we can install Dokku and Docker. Follow the instructions from my previous post.

Once the installation is complete, our Docker containers, deployed applications, and all their config environment variables will be persisted to a transparently encrypted filesystem. Additionally, all our applications log files in /var/log/dokku will also be encrypted (I'll be going over how to get Dokku to send application logs there in a later post).

Destroying Everything

When we want to deprovision the server we can prevent the encryted data from ever being read again by destroying the encryption key and the LUKS header on the block device.

WARNING: Unless you've created a backup of your encryption key and the LUKS header there is no way to undo this operation!

# Shred the encryption key:
$ shred $CRYPTFS_ROOT/key

# Overwrite the beginning of the encrypted block device where the LUKS header is:
$ dd if=/dev/zero of=$CRYPTFS_ROOT/disk bs=1M count=10

Final Thoughts

The biggest hole in all of this is the fact that the encryption key is stored in plaintext alongside the block device file. Unfortunately that's the price to pay for unattended restarts. For a local physical server (ex: your desktop) that is rarely restarted, or one that you have direct console access to, entering the passphrase at each boot is not much of an issue. For a virtual server though, it's an extra step that either needs to be done from the virtual console or via a multi-step boot process (boot, SSH, unlock the encrypted filesystem, resume boot ...).

One solution to this that I've been considering is a hybrid approach. The server would be setup similar to this post, however rather than having the key file stored locally it would be requested at boot time from an external service.

The external service could notify you of the restart (ex: on the service's website or via a push notification to your smart phone). If you approve the request then it would respond with the encryption key, the server would mount the encrypted filesystem, and then resume the boot process. Any request not explicitly approved in X minutes would be automatically rejected. Additionaly once the encrypted filesystem is mounted the server would discard the encryption key (so it would only be in memory while the server remains on).

I think that would strike a good balance between security and usability.

Have an idea of how to improve this setup or have a working version of what I'm describing? Tell me about it.

How to set up a private PaaS with Dokku on DigitalOcean

2014-01-07T00:00:00-05:00

Background

Dokku is a Docker powered mini-Heroku. Once it's installed on a server you can perform git push style deployments to your own private PaaS (Platform as a Service).

In this post we'll go over how to set it up on a $10/mo 1GB DigitalOcean droplet.

Note: At the time of this writing, both Dokku and Docker are still in beta and not meant for production usage.

Domain and Server Provisioning

Register a Domain Name

If you already have a domain name you'll be using, then you can skip this step. If not, then go register one (I use Namecheap).

For the purposes of this post I'll be using the domain for this site "launchbylunch.com".

DNS

Once you have your domain name, edit your name servers to use DigitalOcean's. On Namecheap, once you've clicked on the management page for your domain, you can find the nameserver menu under Domain Name Server Setup on the left.

Alternatively, you can stick to using your registrar's nameservers and setup the wild card CNAME handle in the next section there. Doing it through DigitalOcean is just a bit easier as they'll already have your server details.

Note: Either way it may take up to an hour for your DNS changes to propagate.

SSH

SSH Key Creation

We'll be using SSH to connect to our server and it's a best practice to use SSH keys for authentication rather than passwords. Generate a new pair of SSH keys for use with your DigitalOcean account by running ssh-keygen on your local machine.

It'll prompt you for a name for the SSH key files (specify id_rsa_digital_ocean) and a passphrase (pick something long you can remember) to secure the private key file.

It should look something like this:

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/sehrope/.ssh/id_rsa): id_rsa_digital_ocean
Your identification has been saved in id_rsa_digital_ocean.
Your public key has been saved in id_rsa_digital_ocean.pub.
The key fingerprint is:
e2:44:e3:0b:52:86:40:2a:bd:5d:63:70:a3:88:01:ef sehrope@ls01

I like to keep all my SSH keys in the ~/.ssh directory so to lets move them there:

$ mv id_rsa_digital_ocean* ~/.ssh/.

SSH Config

Normally, to log in to a server using an SSH key you need to specify it on the command line via -i. Rather than specifying it everytime, you can also add default SSH settings to your SSH config file, ~/.ssh/config.

Edit the file (or create it if it doesn't exist) and add an entry for your domain name:

Host launchbylunch.com
  User root
  IdentityFile ~/.ssh/id_rsa_digital_ocean

Now, whenever I SSH to launchbylunch.com it'll default to using that specific SSH key file for authentication. I've also specified the default user to connect to on the remote server.

As an added bonus, if you have shell completions setup properly then you can just type ssh l and then hit tab and it'll autocomplete server names for you.

DigitalOcean

If you haven't done so already, create an account with DigitalOcean.

Next, from the account menu click SSH Keys. Then pick and name for your key and copy/paste the public key file (~/.ssh/id_rsa_digital_ocean.pub) into the text box and save it.

Create a new droplet

The default size is 512mb, change it to 1GB
If you'd like pick a region that is closer to you.
Select the Ubuntu 13.04 x64 image
Click to select the SSH key you added.
Click Create Droplet

In about a minute the droplet should be created. Make a note of its IP address.

DigitalOcean DNS

The final part of our setup will be to setup a wildcard DNS entry for our domain.

From the main DigitalOcean menu click DNS then:

Enter your domain (ex: launchbylunch.com)
Click to select the droplet you just created
Click CREATE DOMAIN

This will create a DNS A record for your root domain pointing to your newly created droplet. To allow apps to be hosted on the sub domain, we'll need to add a wildcard CNAME entry. From the DNS menu of your domain:

Click Add Record
For Record Type select CNAME
For the Name enter *
For the Hostname enter @
Click CREATE

Once you've completed this step it should look something like this:

Connect to the droplet

Now let's see if everything is setup properly. From the command line try to connect to it via SSH:

$ ssh launchbylunch.com

If everything is setup properly then you should be connected. If not, then it's possible your DNS has not propagated yet. Use either nslookup or dig to verify if your DNS has propagated:

$ nslookup launchbylunch.com
Non-authoritative answer:
Name: launchbylunch.com
Address: 162.243.228.51

To check the wildcard CNAME, prefix anything in front of your domain:

$ nslookup foobar.launchbylunch.com
Non-authoritative answer:
foobar.launchbylunch.com  canonical name = launchbylunch.com.
Name: launchbylunch.com
Address: 162.243.228.51

Server Setup

Update System Packages

Now that you're connected to your server first update and upgrade your system packages. If you followed the instructions thus far then you'll be logged in as root and will not need to prefix your commands with sudo.

To update your droplet run:

$ apt-get update && apt-get -y upgrade

After the first update I like to reboot the server once to make sure everything is working fine. From the command line run:

$ reboot

Install Dokku & Docker

Once your server reboots connect again via SSH. We'll now be installing the latest development version of Dokku directly from GitHub:

# Copy the installation script locally:
$ wget -qO- https://raw.github.com/progrium/dokku/master/bootstrap.sh > dokku-setup.sh    
# Run it:
$ bash dokku-setup.sh

This will take a few minutes. Once it completes, Docker and Dokku will be installed on your droplet and a dokku user will have been created. This dokku user will handle your app deployments.

Next, from your local machine (not the droplet terminal) add your SSH key to the remote dokku user. The last parameter me@mycomp is a descriptive name so put whatever you'd like there.

$ cat ~/.ssh/id_rsa_digital_ocean.pub | ssh launchbylunch.com "sudo sshcommand acl-add dokku me@mycomp"

Sample Application

Create

To test out your new private PaaS let's create a project. We'll use the Heroku Node.js sample application. Run the following on your local machine to clone it:

$ git clone https://github.com/heroku/node-js-sample
$ cd node-js-sample

Then, add your new Dokku powered PaaS as a git remote:

$ git remote add production dokku@launchbylunch.com:my-new-app

Deploy

Finally, to push your application to your private PaaS:

$ git push production master

If everything is setup properly, then Dokku should receive the git push and start building your app. Once it's completed it will print something like this:

-----> Deploying my-new-app ...
=====> Application deployed:
       http://my-new-app.launchbylunch.com

To dokku@launchbylunch.com:my-new-app
 * [new branch]      master -> master

Run

If everything deployed properly, then you should be able to test out your newly deployed app.

Open the link for your app an you should see Hello World!.

To deploy a new version, simply commit it locally and push the changes again via git.

Logs

By default Dokku apps send all logging output to stdout. This gets saved by Docker to a single file per container. To see your application logs, SSH to your server and run:

$ dokku logs my-new-app

More information

In a later post I'll go into a Dokku plugin that allows you to run multiple process types as well as a better way to deal with logging.

Are you building something cool with Docker or Dokku? Tell me about it.

New Year, New Blog

2014-01-01T00:00:00-05:00

This site will serve as my new blog.

I'll be writing about databases, app development, devops, and everything in between.

The blog itself is static site generated via Pelican, deployed via Dokku (into a Docker container), served via nginx, and it all runs on a DigitalOcean droplet.