Skip to content

Commit

Permalink
Merge branch 'main' into PG-1095-check-workflow
Browse files Browse the repository at this point in the history
  • Loading branch information
@artemgavrilov
artemgavrilov authored Nov 8, 2024
2 parents c449130 + 2f31c0e commit 3308b82
Show file tree
Hide file tree
Showing 11 changed files with 262 additions and 314 deletions.
274 changes: 129 additions & 145 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,165 +1,167 @@
# pg_tde
[![Forum](https://img.shields.io/badge/Forum-join-brightgreen)](https://forums.percona.com/)

This is an `experimental` encrypted access method for PostgreSQL 16. [We need your feedback!](https://github.com/percona/pg_tde/discussions/151)
# pg_tde: Transparent Database Encryption for PostgreSQL

## Latest test release

To download the latest build of the main branch, use the `HEAD` release from [releases](https://github.com/percona/pg_tde/releases).

Builds are available in a tar.gz format, containing only the required files, and as a deb package.
The deb package is built against the pgdg16 release, but this dependency is not yet enforced in the package.
The PostgreSQL extension provides data at rest encryption. It is currently in an experimental phase and is under active development. [We need your feedback!](https://github.com/percona/pg_tde/discussions/151)

## Documentation
## Table of contents
1. [Overview](#overview)
2. [Documentation](#documentation)
1. [Percona Server for PostgreSQL](#percona-server-for-postgresql)
3. [Build from sources](#building-from-sources-for-community-postgresql)
4. [Run in docker](#run-in-docker)
5. [Setting up](#setting-up)
6. [Helper functions](#helper-functions)

Find more information about `pg_tde` in the [documentation](https://percona.github.io/pg_tde/).
## Overview
Transparent Data Encryption offers encryption at the file level and solves the problem of protecting data at rest. The encryption is transparent for users allowing them to access and manipulate the data and not to worry about the encryption process. As a key provider, the extension supports the keyringfile and [Hashicorp Vault](https://www.vaultproject.io/).

## Installation steps
### This extension provides two `access methods` with different options:

1. Build and install the plugin with make [from source](#build-from-source), or download a [release](https://github.com/percona/pg_tde/releases) and [install the package](#install-from-package)
2. `pg_tde` needs to be loaded at the start time. The extension requires additional shared memory; therefore, add the `pg_tde` value for the `shared_preload_libraries` parameter and restart the `postgresql` instance.
#### `tde_heap_basic` access method
- Works with community PostgreSQL 16 and 17 or with [Percona Server for PosgreSQL 17](https://docs.percona.com/postgresql/17/postgresql-server.html)
- Encrypts tuples and WAL
- **Doesn't** encrypt indexes, temporary files, statistics
- CPU expensive as it decrypts pages each time they are read from bufferpool

Use the [ALTER SYSTEM](https://www.postgresql.org/docs/current/sql-altersystem.html) command from `psql` terminal to modify the `shared_preload_libraries` parameter.
#### `tde_heap` access method
- Works only with [Percona Server for PostgreSQL 17](https://docs.percona.com/postgresql/17/postgresql-server.html)
- Uses extended Storage Manager and WAL APIs
- Encrypts tuples, WAL and indexes
- **Doesn't** encrypt temporary files and statistics **yet**
- Faster and cheaper than `tde_heap_basic`

```sql
ALTER SYSTEM SET shared_preload_libraries = 'pg_tde';
```
## Documentation

3. Start or restart the `postgresql` instance to apply the changes.
Full and comprehensive documentation about `pg_tde` is available at https://percona.github.io/pg_tde/.

* On Debian and Ubuntu:
## Percona Server for PostgreSQL

```sh
sudo systemctl restart postgresql.service
```
Percona provides binary packages of `pg_tde` extension only for Percona Server for PostgreSQL. Learn how to install them or build `pg_tde` from sources for PSPG in the [documentation](https://percona.github.io/pg_tde/main/install.html).

* On RHEL 8 compatible OS:
```sh
sudo systemctl restart postgresql-16.service
```
## Building from sources for community PostgreSQL
1. Install required dependencies (replace XX with 16 or 17)
- On Debian and Ubuntu:
```sh
sudo apt install make gcc autoconf git libcurl4-openssl-dev postgresql-server-dev-XX
```

- On RHEL 8 compatible OS:
```sh
sudo yum install epel-release
yum --enablerepo=powertools install git make gcc autoconf libcurl-devel perl-IPC-Run redhat-rpm-config openssl-devel postgresqlXX-devel
```

4. Create the extension using the [CREATE EXTENSION](https://www.postgresql.org/docs/current/sql-createextension.html) command. Using this command requires the privileges of a superuser or a database owner. Connect to `psql` as a superuser for a database and run the following command:
- On MacOS:
```sh
brew install make autoconf curl gettext postresql@XX
```

```sql
CREATE EXTENSION pg_tde;
```
2. Install or build postgresql 16 or 17
3. If postgres is installed in a non standard directory, set the `PG_CONFIG` environment variable to point to the `pg_config` executable

5. Create a key provider. Currently, `pg_tde` supports `File` and `Vault-V2` key providers. You can add the required key provider using one of the functions.

```sql
-- For Vault-V2 key provider
pg_tde_add_key_provider_vault_v2(
provider_name VARCHAR(128),
vault_token TEXT,
vault_url TEXT,
vault_mount_path TEXT,
vault_ca_path TEXT);

-- For File key provider
FUNCTION pg_tde_add_key_provider_file(
provider_name VARCHAR(128),
file_path TEXT);
4. Clone the repository, build and install it with the following commands:

```
**Example**: Add a `File` key provider and name it `file`.
```sql
SELECT pg_tde_add_key_provider_file('file','/tmp/pgkeyring');
```
**Note: The `File` provided is intended for development and stores the keys unencrypted in the specified data file.**

6. Set the principal key for the database using the `pg_tde_set_principal_key` function.
```sql
FUNCTION pg_tde_set_principal_key (
principal_key_name VARCHAR(255),
provider_name VARCHAR(255));
```
**Example**: Set the principal key named `my-principal-key` using the `file` as a key provider.
```sql
SELECT pg_tde_set_principal_key('my-principal-key','file');
```

7. You are all set to create encrypted tables. For that, specify `USING tde_heap_basic` access method in the `CREATE TABLE` statement.
**For example**:
```sql
CREATE TABLE albums (
album_id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
artist_id INTEGER,
title TEXT NOT NULL,
released DATE NOT NULL
) USING tde_heap_basic;
```
```sh
git clone https://github.com/percona/pg_tde
```

5. Compile and install the extension

## Build from source
```sh
cd pg_tde
./configure
make USE_PGXS=1
sudo make USE_PGXS=1 install
```

1. To build `pg_tde` from source code, you require the following:
## Run in Docker

* On Debian and Ubuntu:
```sh
sudo apt install make gcc autoconf libcurl4-openssl-dev postgresql-server-dev-16
```
There is a [docker image](https://hub.docker.com/r/perconalab/pg_tde) with `pg_tde` based community [PostgreSQL 16](https://hub.docker.com/_/postgres)

* On RHEL 8 compatible OS:
```sh
sudo yum install epel-release
yum --enablerepo=powertools install git make gcc autoconf libcurl-devel postgresql16-devel perl-IPC-Run redhat-rpm-config openssl-devel
```

* On MacOS:
```sh
brew install make autoconf curl gettext postresql@16
docker run --name pg-tde -e POSTGRES_PASSWORD=mysecretpassword -d perconalab/pg_tde
```
Docker file is available [here](https://github.com/percona/pg_tde/blob/main/docker/Dockerfile)
2. Install or build postgresql 16 [(see reference commit below)](#base-commit)
3. If postgres is installed in a non standard directory, set the `PG_CONFIG` environment variable to point to the `pg_config` executable

4. Clone the repository, build and install it with the following commands:
```
git clone https://github.com/percona/pg_tde
```
_See [Make Builds for Developers](https://github.com/percona/pg_tde/wiki/Make-builds-for-developers) for more info on the build infrastructure._
Compile and install the extension
## Setting up
1. Add extension to the `shared_preload_libraries`:
1. Via configuration file `postgresql.conf `
```
shared_preload_libraries=pg_tde
```
2. Via SQL using [ALTER SYSTEM](https://www.postgresql.org/docs/current/sql-altersystem.html) command
```sql
ALTER SYSTEM SET shared_preload_libraries = 'pg_tde';
```
2. Start or restart the `postgresql` instance to apply the changes.
* On Debian and Ubuntu:
```sh
sudo systemctl restart postgresql.service
```
* On RHEL 8 compatible OS (replace XX with your version):
```sh
sudo systemctl restart postgresql-XX.service
```
3. [CREATE EXTENSION](https://www.postgresql.org/docs/current/sql-createextension.html) with SQL (requires superuser or a database owner privileges):
```sql
CREATE EXTENSION pg_tde;
```
4. Create a key provider. Currently `pg_tde` supports `File` and `Vault-V2` key providers. You can add the required key provider using one of the functions.
```
cd pg_tde
./configure
make USE_PGXS=1
sudo make USE_PGXS=1 install
```
On RHEL 8 compatible OS:
```
PATH=$PATH:/usr/pgsql-16/bin/ make USE_PGXS=1
sudo PATH=$PATH:/usr/pgsql-16/bin/ make USE_PGXS=1 install
```
```sql
-- For Vault-V2 key provider
-- pg_tde_add_key_provider_vault_v2(provider_name, vault_token, vault_url, vault_mount_path, vault_ca_path)
SELECT pg_tde_add_key_provider_vault_v2(
'vault-provider',
json_object( 'type' VALUE 'remote', 'url' VALUE 'http://localhost:8888/token' ),
json_object( 'type' VALUE 'remote', 'url' VALUE 'http://localhost:8888/url' ),
to_json('secret'::text), NULL);
_See [Make Builds for Developers](https://github.com/percona/pg_tde/wiki/Make-builds-for-developers) for more info on the build infrastructure._
-- For File key provider
-- pg_tde_add_key_provider_file(provider_name, file_path);
SELECT pg_tde_add_key_provider_file('file','/tmp/pgkeyring');
```
## Install from package
**Note: The `File` provided is intended for development and stores the keys unencrypted in the specified data file.**
1. Download the latest [release package](https://github.com/percona/pg_tde/releases)
5. Set the principal key for the database using the `pg_tde_set_principal_key` function.
``` sh
wget https://github.com/percona/pg_tde/releases/download/latest/pgtde-pgdg16.deb
```
2. Install the package
```sql
-- pg_tde_set_principal_key(principal_key_name, provider_name);
SELECT pg_tde_set_principal_key('my-principal-key','file');
```
6. Specify `tde_heap_basic` access method during table creation
```sql
CREATE TABLE albums (
album_id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
artist_id INTEGER,
title TEXT NOT NULL,
released DATE NOT NULL
) USING tde_heap_basic;
```
7. You can encrypt existing table. It requires rewriting the table, so for large tables, it might take a considerable amount of time.
```sql
ALTER TABLE table_name SET access method tde_heap_basic;
```
``` sh
sudo dpkg -i pgtde-pgdg16.deb
```
## Run in Docker
## Latest test release
You can find docker images built from the current main branch on [Docker Hub](https://hub.docker.com/r/perconalab/pg_tde). Images build on top of [postgres:16](https://hub.docker.com/_/postgres) official image. To run it:
```
docker run --name pg-tde -e POSTGRES_PASSWORD=mysecretpassword -d perconalab/pg_tde
```
It builds and adds `pg_tde` extension to Postgres 16. Relevant `postgresql.conf` and `tde_conf.json` are created in `/etc/postgresql/` inside the container. This dir is exposed as volume.
To download the latest build of the main branch, use the `HEAD` release from [releases](https://github.com/percona/pg_tde/releases).
See https://hub.docker.com/_/postgres on usage.
Builds are available in a tar.gz format, containing only the required files, and as a deb package.
The deb package is built against the pgdg16 release, but this dependency is not yet enforced in the package.
You can also build a docker image manually with:
```
docker build . -f ./docker/Dockerfile -t your-image-name
```
## Helper functions
Expand All @@ -168,21 +170,3 @@ The extension provides the following helper functions:
### pg_tde_is_encrypted(tablename)
Returns `t` if the table is encrypted (uses the tde_heap_basic access method), or `f` otherwise.

## Base commit

This is based on the heap code as of the following commit:

```
commit a81e5516fa4bc53e332cb35eefe231147c0e1749 (HEAD -> REL_16_STABLE, origin/REL_16_STABLE)
Author: Amit Kapila <[email protected]>
Date: Wed Sep 13 09:48:31 2023 +0530
Fix the ALTER SUBSCRIPTION to reflect the change in run_as_owner option.
```

## FAQ

### What would be the best way to encrypt an existing table using pg_tde?

- That can be done by using the ALTER TABLE command and changing the table's access method to tde_heap_basic. `ALTER TABLE table_name SET access method tde_heap_basic;` This command rewrites the table, so for large tables, it might take a considerable amount of time. Unfortunately, rewriting the table is currently the only available option.
2 changes: 1 addition & 1 deletion documentation/_resource/overrides/main.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

{% block announce %}
This is a Beta version of Percona Transparent Encryption extension and it is
<strong>not recommended for production environments</strong> yet. We encourage you to test it and give your feedback.
<strong>not recommended for production environments</strong> yet. We encourage you to test it and <a href= "https://forums.percona.com/c/postgresql/pg-tde-transparent-data-encryption-tde/82">give your feedback</a>.
This will help us improve the product and make it production-ready faster.
{% endblock %}

Expand Down
2 changes: 1 addition & 1 deletion documentation/_resource/overrides/partials/header.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
<path d="m7.825 13 5.6 5.6L12 20l-8-8 8-8 1.425 1.4-5.6 5.6H20v2H7.825Z"/>
</g>
</svg>
<span>Percona Documentation</span>
<span>Percona Software for PostgreSQL Documentation</span>
</a>
</div>
</div>
Expand Down
Loading

0 comments on commit 3308b82

Please sign in to comment.