postgresql/contrib/pgcrypto

pgcrypto - cryptographic functions for PostgreSQL
=================================================
Marko Kreen <marko@l-t.ee>

// Note: this document is in asciidoc format.


1.  Installation
-----------------

Run following commands:

    make
    make install
    make installcheck

The `make installcheck` command is important.  It runs regression tests
for the module.  They make sure the functions here produce correct
results.

Next, to put the functions into a particular database, run the commands in
file pgcrypto.sql, which has been installed into the shared files directory.

Example using psql:

    psql -d DBNAME -f pgcrypto.sql


2.  Notes
----------

2.1.  Configuration
~~~~~~~~~~~~~~~~~~~~

pgcrypto configures itself according to the findings of main PostgreSQL
`configure` script.  The options that affect it are `--with-zlib` and
`--with-openssl`.

When compiled with zlib, PGP encryption functions are able to
compress data before encrypting.

When compiled with OpenSSL there will be more algorithms available.
Also public-key encryption functions will be faster as OpenSSL
has more optimized BIGNUM functions.

Summary of functionality with and without OpenSSL:

`----------------------------`---------`------------
 Functionality                built-in   OpenSSL
----------------------------------------------------
 MD5                          yes       yes
 SHA1                         yes       yes
 SHA224/256/384/512           yes       yes (3)
 Any other digest algo        no        yes (1)
 Blowfish                     yes       yes
 AES                          yes       yes (2)
 DES/3DES/CAST5               no        yes
 Raw encryption               yes       yes
 PGP Symmetric encryption     yes       yes
 PGP Public-Key encryption    yes       yes
----------------------------------------------------

1. Any digest algorithm OpenSSL supports is automatically picked up.
   This is not possible with ciphers, which need to be supported
   explicitly.

2. AES is included in OpenSSL since version 0.9.7.  If pgcrypto is
   compiled against older version, it will use built-in AES code,
   so it has AES always available.

3. SHA2 algorithms were added to OpenSSL in version 0.9.8.  For
   older versions, pgcrypto will use built-in code.


2.2.  NULL handling
~~~~~~~~~~~~~~~~~~~~

As standard in SQL, all functions return NULL, if any of the arguments
are NULL.  This may create security risks on careless usage.


2.3.  Deprecated functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The `digest_exists()`, `hmac_exists()` and `cipher_exists()` functions
are deprecated.  The plan is to remove them in PostgreSQL 8.2.


2.4.  Security
~~~~~~~~~~~~~~~

All the functions here run inside database server.  That means that all
the data and passwords move between pgcrypto and client application in
clear-text.  Thus you must:

1.  Connect locally or use SSL connections.
2.  Trust both system and database administrator.

If you cannot, then better do crypto inside client application.


3.  General hashing
--------------------

3.1.  digest(data, type)
~~~~~~~~~~~~~~~~~~~~~~~~~

  digest(data text, type text) RETURNS bytea
  digest(data bytea, type text) RETURNS bytea

Type is here the algorithm to use.  Standard algorithms are `md5` and
`sha1`, although there may be more supported, depending on build
options.

Returns binary hash.

If you want hexadecimal string, use `encode()` on result.  Example:

    CREATE OR REPLACE FUNCTION sha1(bytea) RETURNS text AS $$
      SELECT encode(digest($1, 'sha1'), 'hex')
    $$ LANGUAGE SQL STRICT IMMUTABLE;


3.2.  hmac(data, key, type)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  hmac(data text, key text, type text) RETURNS bytea
  hmac(data bytea, key text, type text) RETURNS bytea

Calculates Hashed MAC over data.  `type` is the same as in `digest()`.
If the key is larger than hash block size it will first hashed and the
hash will be used as key.

It is similar to digest() but the hash can be recalculated only knowing
the key.  This avoids the scenario of someone altering data and also
changing the hash.

Returns binary hash.



4.  Password hashing
---------------------

The functions `crypt()` and `gen_salt()` are specifically designed
for hashing passwords.  `crypt()` does the hashing and `gen_salt()`
prepares algorithm parameters for it.

The algorithms in `crypt()` differ from usual hashing algorithms like
MD5 or SHA1 in following respects:

1. They are slow.  As the amount of data is so small, this is only
   way to make brute-forcing passwords hard.
2. Include random 'salt' with result, so that users having same
   password would have different crypted passwords.  This is also
   additional defense against reversing the algorithm.
3. Include algorithm type in the result, so passwords hashed with
   different algorithms can co-exist.
4. Some of them are adaptive - that means after computers get
   faster, you can tune the algorithm to be slower, without
   introducing incompatibility with existing passwords.

Supported algorithms:
`------`-------------`---------`----------`---------------------------
 Type   Max password  Adaptive  Salt bits  Description
----------------------------------------------------------------------
`bf`     72           yes         128      Blowfish-based, variant 2a
`md5`    unlimited    no           48      md5-based crypt()
`xdes`   8            yes          24      Extended DES
`des`    8            no           12      Original UNIX crypt
----------------------------------------------------------------------


4.1.  crypt(password, salt)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  crypt(password text, salt text) RETURNS text

Calculates UN*X crypt(3) style hash of password.  When storing new
password, you need to use function `gen_salt()` to generate new salt.
When checking password you should use existing hash as salt.

Example - setting new password:

    UPDATE .. SET pswhash = crypt('new password', gen_salt('md5'));

Example - authentication:

    SELECT pswhash = crypt('entered password', pswhash) WHERE .. ;

returns true or false whether the entered password is correct.
It also can return NULL if `pswhash` field is NULL.


4.2.  gen_salt(type)
~~~~~~~~~~~~~~~~~~~~~

  gen_salt(type text) RETURNS text

Generates a new random salt for usage in `crypt()`.  For adaptible
algorithms, it uses the default iteration count.

Accepted types are: `des`, `xdes`, `md5` and `bf`.


4.3.  gen_salt(type, rounds)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  gen_salt(type text, rounds integer) RETURNS text

Same as above, but lets user specify iteration count for some
algorithms.  The higher the count, the more time it takes to hash
the password and therefore the more time to break it.  Although with
too high count the time to calculate a hash may be several years
- which is somewhat impractical.

Number is algorithm specific:

`-----'---------'-----'----------
 type   default   min   max
---------------------------------
 `xdes`     725     1   16777215
 `bf`         6     4         31
---------------------------------

In case of xdes there is a additional limitation that the count must be
a odd number.

Notes:

- Original DES crypt was designed to have the speed of 4 hashes per
  second on the hardware of that time.
- Slower than 4 hashes per second would probably dampen usability.
- Faster than 100 hashes per second is probably too fast.
- See next section about possible values for `crypt-bf`.


4.4.  Comparison of crypt and regular hashes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Here is a table that should give overview of relative slowness
of different hashing algorithms.

* The goal is to crack a 8-character password, which consists:
  1.  Only of lowercase letters
  2.  Numbers, lower- and uppercase letters.
* The table below shows how much time it would take to try all
  combinations of characters.
* The `crypt-bf` is featured in several settings - the number
  after slash is the `rounds` parameter of `gen_salt()`.

`------------'----------'--------------'--------------------
Algorithm     Hashes/sec  Chars: [a-z]   Chars: [A-Za-z0-9]
------------------------------------------------------------
crypt-bf/8            28     246 years         251322 years
crypt-bf/7            57     121 years         123457 years
crypt-bf/6           112      62 years          62831 years
crypt-bf/5           211      33 years          33351 years
crypt-md5           2681     2.6 years           2625 years
crypt-des         362837        7 days             19 years
sha1              590223        4 days             12 years
md5              2345086         1 day              3 years
------------------------------------------------------------

* The machine used is 1.5GHz Pentium 4.
* crypt-des and crypt-md5 algorithm numbers are taken from
  John the Ripper v1.6.38 `-test` output.
* MD5 numbers are from mdcrack 1.2.
* SHA1 numbers are from lcrack-20031130-beta.
* `crypt-bf` numbers are taken using simple program that loops
  over 1000 8-character passwords.  That way I can show the speed with
  different number of rounds.  For reference: `john -test` shows 213
  loops/sec for crypt-bf/5.  (The small difference in results is in
  accordance to the fact that the `crypt-bf` implementation in pgcrypto
  is same one that is used in John the Ripper.)

Note that "try all combinations" is not a realistic exercise.
Usually password cracking is done with the help of dictionaries, which
contain both regular words and various mutations of them.  So, even
somewhat word-like passwords could be cracked much faster than the above
numbers suggest, and a 6-character non-word like password may escape
cracking.  Or not.


5.  PGP encryption
-------------------

The functions here implement the encryption part of OpenPGP (RFC2440)
standard.   Supported are both symmetric-key and public-key encryption.


5.1.  Overview
~~~~~~~~~~~~~~~

Encrypted PGP message consists of 2 packets:

- Packet for session key - either symmetric- or public-key encrypted.
- Packet for session-key encrypted data.

When encrypting with password:

1. Given password is hashed using String2Key (S2K) algorithm.  This
   is rather similar to `crypt()` algorithm - purposefully slow
   and with random salt - but it produces a full-length binary key.
2. If separate session key is requested, new random key will be
   generated.  Otherwise S2K key will be used directly as session key.
3. If S2K key is to be used directly, then only S2K settings will be put
   into session key packet.  Otherwise session key will be encrypted with
   S2K key and put into session key packet.

When encrypting with public key:

1. New random session key is generated.
2. It is encrypted using public key and put into session key packet.

Now common part, the session-key encrypted data packet:

1. Optional data-manipulation: compression, conversion to UTF-8,
   conversion of line-endings.
2. Data is prefixed with block of random bytes.  This is equal
   to using random IV.
3. A SHA1 hash of random prefix and data is appended.
4. All this is encrypted with session key.


5.2.  pgp_sym_encrypt(data, psw)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  pgp_sym_encrypt(data text, psw text [, options text] ) RETURNS bytea
  pgp_sym_encrypt_bytea(data bytea, psw text [, options text] ) RETURNS bytea

Return a symmetric-key encrypted PGP message.

Options are described in section 5.8.


5.3. pgp_sym_decrypt(msg, psw)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  pgp_sym_decrypt(msg bytea, psw text [, options text] ) RETURNS text
  pgp_sym_decrypt_bytea(msg bytea, psw text [, options text] ) RETURNS bytea

Decrypt a symmetric-key encrypted PGP message.

Decrypting bytea data with `pgp_sym_decrypt` is disallowed.
This is to avoid outputting invalid character data.  Decrypting
originally textual data with `pgp_sym_decrypt_bytea` is fine.

Options are described in section 5.8.


5.4.  pgp_pub_encrypt(data, pub_key)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  pgp_pub_encrypt(data text, key bytea [, options text] ) RETURNS bytea
  pgp_pub_encrypt_bytea(data bytea, key bytea [, options text] ) RETURNS bytea

Encrypt data with a public key.  Giving this function a secret key will
produce a error.

Options are described in section 5.8.


5.5.  pgp_pub_decrypt(msg, sec_key [, psw])
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  pgp_pub_decrypt(msg bytea, key bytea [, psw text [, options text]] ) \
  RETURNS text
  pgp_pub_decrypt_bytea(msg bytea, key bytea [,psw text [, options text]] ) \
  RETURNS bytea

Decrypt a public-key encrypted message with secret key.  If the secret
key is password-protected, you must give the password in `psw`.  If
there is no password, but you want to specify option for function, you
need to give empty password.

Decrypting bytea data with `pgp_pub_decrypt` is disallowed.
This is to avoid outputting invalid character data.  Decrypting
originally textual data with `pgp_pub_decrypt_bytea` is fine.

Options are described in section 5.8.


5.6.  pgp_key_id(key / msg)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  pgp_key_id(key or msg bytea) RETURNS text

It shows you either key ID if given PGP public or secret key.  Or it
gives the key ID that was used for encrypting the data, if given
encrypted message.

It can return 2 special key IDs:

SYMKEY::
   The data is encrypted with symmetric key.

ANYKEY::
   The data is public-key encrypted, but the key ID is cleared.
   That means you need to try all your secret keys on it to see
   which one decrypts it.  pgcrypto itself does not produce such
   messages.

Note that different keys may have same ID.   This is rare but normal
event.  Client application should then try to decrypt with each one,
to see which fits - like handling ANYKEY.


5.7.  armor / dearmor
~~~~~~~~~~~~~~~~~~~~~~

  armor(data bytea) RETURNS text
  dearmor(data text) RETURNS bytea

Those wrap/unwrap data into PGP Ascii Armor which is basically Base64
with CRC and additional formatting.


5.8.  Options for PGP functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Options are named to be similar to GnuPG.  Values should be given after
an equal sign; separate options from each other with commas.  Example:

  pgp_sym_encrypt(data, psw, 'compress-algo=1, cipher-algo=aes256')

All of the options except `convert-crlf` apply only to encrypt
functions.  Decrypt functions get the parameters from PGP data.

Most interesting options are probably `compression-algo` and
`unicode-mode`.  The rest should have reasonable defaults.


cipher-algo::
  What cipher algorithm to use.

  Values: bf, aes128, aes192, aes256 (OpenSSL-only: `3des`, `cast5`)
  Default: aes128
  Applies: pgp_sym_encrypt, pgp_pub_encrypt

compress-algo::
  Which compression algorithm to use.  Needs building with zlib.

  Values:
    0 - no compression
    1 - ZIP compression
    2 - ZLIB compression [=ZIP plus meta-data and block-CRC's]
  Default: 0
  Applies: pgp_sym_encrypt, pgp_pub_encrypt

compress-level::
  How much to compress.  Bigger level compresses smaller but is slower.
  0 disables compression.

  Values: 0, 1-9
  Default: 6
  Applies: pgp_sym_encrypt, pgp_pub_encrypt

convert-crlf::
  Whether to convert `\n` into `\r\n` when encrypting and `\r\n` to `\n`
  when decrypting.  RFC2440 specifies that text data should be stored
  using `\r\n` line-feeds.  Use this to get fully RFC-compliant
  behavior.

  Values: 0, 1
  Default: 0
  Applies: pgp_sym_encrypt, pgp_pub_encrypt, pgp_sym_decrypt, pgp_pub_decrypt

disable-mdc::
  Do not protect data with SHA-1.  Only good reason to use this
  option is to achieve compatibility with ancient PGP products, as the
  SHA-1 protected packet is from upcoming update to RFC2440.  (Currently
  at version RFC2440bis-14.) Recent gnupg.org and pgp.com software
  supports it fine.

  Values: 0, 1
  Default: 0
  Applies: pgp_sym_encrypt, pgp_pub_encrypt

enable-session-key::
  Use separate session key.  Public-key encryption always uses separate
  session key, this is for symmetric-key encryption, which by default
  uses S2K directly.

  Values: 0, 1
  Default: 0
  Applies: pgp_sym_encrypt

s2k-mode::
  Which S2K algorithm to use.

  Values:
    0 - Without salt.  Dangerous!
    1 - With salt but with fixed iteration count.
    3 - Variable iteration count.
  Default: 3
  Applies: pgp_sym_encrypt

s2k-digest-algo::
  Which digest algorithm to use in S2K calculation.

  Values: md5, sha1
  Default: sha1
  Applies: pgp_sym_encrypt

s2k-cipher-algo::
  Which cipher to use for encrypting separate session key.

  Values: bf, aes, aes128, aes192, aes256
  Default: use cipher-algo.
  Applies: pgp_sym_encrypt

unicode-mode::
  Whether to convert textual data from database internal encoding to
  UTF-8 and back.  If your database already is UTF-8, no conversion will
  be done, only the data will be tagged as UTF-8.  Without this option
  it will not be.

  Values: 0, 1
  Default: 0
  Applies: pgp_sym_encrypt, pgp_pub_encrypt


5.9.  Generating keys with GnuPG
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Generate a new key:

    gpg --gen-key

The preferred key type is "DSA and Elgamal".

For RSA encryption you must create either DSA or RSA sign-only key
as master and then add RSA encryption subkey with `gpg --edit-key`.

List keys:

    gpg --list-secret-keys

Export ascii-armored public key:

    gpg -a --export KEYID > public.key

Export ascii-armored secret key:

    gpg -a --export-secret-keys KEYID > secret.key

You need to use `dearmor()` on them before giving them to
pgp_pub_* functions.  Or if you can handle binary data, you can drop
"-a" from gpg.

For more details see `man gpg`, http://www.gnupg.org/gph/en/manual.html[
The GNU Privacy Handbook] and other docs on http://www.gnupg.org[] site.


5.10.  Limitations of PGP code
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- No support for signing.  That also means that it is not checked
  whether the encryption subkey belongs to master key.

- No support for encryption key as master key.  As such practice
  is generally discouraged, it should not be a problem.

- No support for several subkeys.  This may seem like a problem, as this
  is common practice.  On the other hand, you should not use your regular
  GPG/PGP keys with pgcrypto, but create new ones, as the usage scenario
  is rather different.


6.  Raw encryption
-------------------

Those functions only run a cipher over data, they don't have any advanced
features of PGP encryption.  Therefore they have some major problems:

1. They use user key directly as cipher key.
2. They don't provide any integrity checking, to see
   if the encrypted data was modified.
3. They expect that users manage all encryption parameters
   themselves, even IV.
4. They don't handle text.

So, with the introduction of PGP encryption, usage of raw
encryption functions is discouraged.


    encrypt(data bytea, key bytea, type text) RETURNS bytea
    decrypt(data bytea, key bytea, type text) RETURNS bytea

    encrypt_iv(data bytea, key bytea, iv bytea, type text) RETURNS bytea
    decrypt_iv(data bytea, key bytea, iv bytea, type text) RETURNS bytea

Encrypt/decrypt data with cipher, padding data if needed.

`type` parameter description in pseudo-noteup:

    algo ['-' mode] ['/pad:' padding]

Supported algorithms:

* `bf`		- Blowfish
* `aes`		- AES (Rijndael-128)

Modes:

* `cbc` - next block depends on previous. (default)
* `ecb` - each block is encrypted separately.
          (for testing only)

Padding:

* `pkcs` - data may be any length (default)
* `none` - data must be multiple of cipher block size.

IV is initial value for mode, defaults to all zeroes.  It is ignored for
ECB.  It is clipped or padded with zeroes if not exactly block size.

So, example:

	encrypt(data, 'fooz', 'bf')

is equal to

	encrypt(data, 'fooz', 'bf-cbc/pad:pkcs')


7.  Random bytes
-----------------

    gen_random_bytes(count integer)

Returns `count` cryptographically strong random bytes as bytea value.
There can be maximally 1024 bytes extracted at a time.  This is to avoid
draining the randomness generator pool.


8.  Credits
------------

I have used code from following sources:

`--------------------`-------------------------`-------------------------------
  Algorithm            Author                    Source origin
-------------------------------------------------------------------------------
  DES crypt()          David Burren and others   FreeBSD libcrypt
  MD5 crypt()          Poul-Henning Kamp         FreeBSD libcrypt
  Blowfish crypt()     Solar Designer            www.openwall.com
  Blowfish cipher      Niels Provos              OpenBSD sys/crypto
  Rijndael cipher      Brian Gladman             OpenBSD sys/crypto
  MD5 and SHA1         WIDE Project              KAME kame/sys/crypto
  SHA256/384/512       Aaron D. Gifford          OpenBSD sys/crypto
  BIGNUM math          Michael J. Fromberger     dartmouth.edu/~sting/sw/imath
-------------------------------------------------------------------------------


9.  Legalese
-------------

* I owe a beer to Poul-Henning.
* This product includes software developed by Niels Provos.


10.  References/Links
----------------------

10.1.  Useful reading
~~~~~~~~~~~~~~~~~~~~~~

http://www.gnupg.org/gph/en/manual.html[]::
	The GNU Privacy Handbook

http://www.openwall.com/crypt/[]::
	Describes the crypt-blowfish algorithm.

http://www.stack.nl/~galactus/remailers/passphrase-faq.html[]::
	How to choose good password.

http://world.std.com/~reinhold/diceware.html[]::
	Interesting idea for picking passwords.

http://www.interhack.net/people/cmcurtin/snake-oil-faq.html[]::
	Describes good and bad cryptography.


10.2.  Technical references
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

http://www.ietf.org/rfc/rfc2440.txt[]::
	OpenPGP message format

http://www.imc.org/draft-ietf-openpgp-rfc2440bis[]::
	New version of RFC2440.

http://www.ietf.org/rfc/rfc1321.txt[]::
	The MD5 Message-Digest Algorithm

http://www.ietf.org/rfc/rfc2104.txt[]::
	HMAC: Keyed-Hashing for Message Authentication

http://www.usenix.org/events/usenix99/provos.html[]::
	Comparison of crypt-des, crypt-md5 and bcrypt algorithms.

http://csrc.nist.gov/cryptval/des.htm[]::
	Standards for DES, 3DES and AES.

http://en.wikipedia.org/wiki/Fortuna_(PRNG)[]::
	Description of Fortuna CSPRNG.

http://jlcooke.ca/random/[]::
	Jean-Luc Cooke Fortuna-based /dev/random driver for Linux.

http://www.cs.ut.ee/~helger/crypto/[]::
	Collection of cryptology pointers.


// $PostgreSQL: pgsql/contrib/pgcrypto/README.pgcrypto,v 1.17 2006/08/05 00:29:11 neilc Exp $