From 77bf42550f1911885e869a3255458d17b4622670 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Wed, 13 Nov 2024 14:04:22 +0100 Subject: [PATCH 1/5] Rebased on main --- documentation/docs/apt.md | 11 +-- documentation/docs/table-access-method.md | 90 +++++++++++++++++++++++ documentation/docs/test.md | 20 ++--- documentation/mkdocs.yml | 5 +- 4 files changed, 107 insertions(+), 19 deletions(-) create mode 100644 documentation/docs/table-access-method.md diff --git a/documentation/docs/apt.md b/documentation/docs/apt.md index 02d87891..7b0fcfbc 100644 --- a/documentation/docs/apt.md +++ b/documentation/docs/apt.md @@ -37,16 +37,10 @@ You need the `percona-release` repository management tool that enables the desir 4. Enable the Percona Distribution for PostgreSQL repository - Percona provides [two repositories](repo-overview.md) for Percona Distribution for PostgreSQL. We recommend enabling the Major release repository to timely receive the latest updates. + Percona provides [two repositories](repo-overview.md) for Percona Distribution for PostgreSQL. We recommend enabling the Major release repository to timely receive the latest updates. Since the `tde_heap` access method is still in the experimental stage, the `pg_tde` package is currently available from the experimental repository. ```{.bash data-prompt="$"} - $ sudo percona-release setup ppg-17 - ``` - -5. Enable the experimental Percona Distribution for PostgreSQL repository that contains the pg_tde package - - ```bash - sudo percona-release enable ppg-{{pgversion17}} experimental + $ sudo percona-release enable ppg-{{pgversion17}} experimental ``` 6. Update the local cache @@ -57,6 +51,7 @@ You need the `percona-release` repository management tool that enables the desir ## Install `pg_tde` +After all [preconditions](#preconditions) are met, install the extension. 1. Install Percona Distribution for PostgreSQL. diff --git a/documentation/docs/table-access-method.md b/documentation/docs/table-access-method.md new file mode 100644 index 00000000..242b2bc5 --- /dev/null +++ b/documentation/docs/table-access-method.md @@ -0,0 +1,90 @@ +# Table access method + +A table access method is the way how PostgreSQL stores the data in a table. The default table access method is `heap`. PostgreSQL organizes data in a heap structure, meaning there is no particular order to the rows in the table. Each row is stored independently, and rows are identified by their unique row identifier (TID). + +## How does the heap access method work? + +**Insertion**: When a new row is inserted, PostgreSQL finds a free space in the tablespace and stores the row there. + +**Deletion**: When a row is deleted, PostgreSQL marks the space occupied by the row as free, but the data remains until it is overwritten by a new insertion. + +**Updates**: Updates are handled by deleting the old row and inserting a new row with the updated values + +## Custom access method + +You can create a custom table access method for each table and instruct PostgreSQL how to store the data for you. For example, you can tailor the table access method to better suit your specific workload or data access patterns. + +To define an access method, use the `CREATE ACCESS METHOD` with the `TYPE` clause set to `table`: + +```sql +CREATE ACCESS METHOD access_method_name TYPE table; +``` + +To use your access method, specify the `USING` clause for the `CREATE TABLE` command: + +```sql +CREATE TABLE table_name ( + column1 data_type, + column2 data_type, + ... +) USING access_method_name; +``` + +## `tde_heap` access method + +The `tde_heap` is a custom table access method that comes with the `pg_tde` extension to provide data encryption. It is automatically created **only** for the databases where you [enabled the `pg_tde` extension](setup.md) and configured the key provider. + + +## Changing the default table access method + +You can change the default table access method so that every table in the entire database cluster is created using the custom access method. For example, you can enable data encryption by default by defining either `tde_heap_basic` or the `tde_heap` as the default table access method. + +However, consider the following before doing so: + +* This is a global setting and applies across the entire database cluster and not just a single database. We recommend setting it with caution only if you created the `pg_tde` extensions for all databases. Otherwise PostgreSQL throws an error. +* You must create the `pg_tde` extension and configure the key provider for all databases before you modify the configuration. Otherwise PostgreSQL won't find the specified access method and will throw an error. + +Here's how you can set the new default table access method: + +1. Add the access method to the `default_table_access_method` parameter. + + === "via the SQL statement" + + Use the `ALTER SYSTEM SET` command. This requires superuser privileges. + + This example shows how to set the `tde_heap` access method. Replace it with the `tde_heap_basic` if needed. + + + ```sql + ALTER SYSTEM SET default_table_access_method=tde_heap; + ``` + + === "via the configuration file" + + Edit the `postgresql.conf` configuration file and add the value for the `default_table_access_method` parameter. + + This example shows how to set the `tde_heap` access method. Replace it with the `tde_heap_basic` if needed. + + ```ini + default_table_access_method = 'tde_heap' + ``` + + === "via the SET command" + + You can use the SET command to change the default table access method temporarily, for the current session. + + Unlike modifying the `postgresql.conf` file or using the ALTER SYSTEM SET command, the changes you make via the SET command don't persist after the session ends. + + You also don't need to have the superuser privileges to run the SET command. + + You can run the SET command anytime during the session. This example shows how to set the `tde_heap` access method. Replace it with the `tde_heap_basic` if needed. + + ```sql + SET default_table_access_method = tde_heap; + ``` + +2. Reload the configuration to apply the changes: + + ```sql + SELECT pg_reload_conf(); + diff --git a/documentation/docs/test.md b/documentation/docs/test.md index 6ed97153..8ad77891 100644 --- a/documentation/docs/test.md +++ b/documentation/docs/test.md @@ -1,16 +1,14 @@ # Test Transparent Data Encryption -To check if the data is encrypted, do the following: +Enabling `pg_tde` extension for a database creates the table access method `tde_heap` . This access method enables you to encrypt the data. -=== "pg_tde Tech preview" +!!! warning - !!! warning + This is the tech preview functionality. Its scope is not yet finalized and can change anytime. **Use it only for testing purposes.** - This is the tech preview functionality. Its scope is not yet finalized and can change anytime.** Use it only for testing purposes.** +Here's how to do it: -To check if the data is encrypted, do the following: - -1. Create a table in the database for which you have [enabled `pg_tde`](setup.md). Enabling `pg_tde` extension creates the table access method `tde_heap`. To enable data encryption, create the table using this access method as follows: +1. Create a table in the database for which you have [enabled `pg_tde`](setup.md) using the `tde_heap` access method as follows: ```sql CREATE TABLE ( ) USING tde_heap; @@ -26,8 +24,10 @@ To check if the data is encrypted, do the following: released DATE NOT NULL ) USING tde_heap; ``` + + Learn more about table access methods and how you can enable data encryption by default in the [Table access methods](table-access-method.md) section. -2. Run the following function: +2. To check if the data is encrypted, run the following function: ```sql SELECT pg_tde_is_encrypted('table_name'); @@ -45,10 +45,10 @@ To check if the data is encrypted, do the following: SELECT pg_tde_rotate_principal_key('new-principal-key', 'new-provider'); -- changeprovider ``` -4. You can encrypt existing table. It requires rewriting the table, so for large tables, it might take a considerable amount of time. +4. You can encrypt an existing table. It requires rewriting the table, so for large tables, it might take a considerable amount of time. ```sql - ALTER TABLE table_name SET access method tde_heap; + ALTER TABLE table_name SET access method tde_heap; ``` !!! hint diff --git a/documentation/mkdocs.yml b/documentation/mkdocs.yml index 8f9a4c5f..4334f73a 100644 --- a/documentation/mkdocs.yml +++ b/documentation/mkdocs.yml @@ -155,7 +155,6 @@ extra: nav: - Home: index.md - - tde.md - Get started: - "Install": "install.md" - "Via apt": apt.md @@ -163,6 +162,10 @@ nav: - "Set up": "setup.md" - "Test TDE": "test.md" - functions.md + - Concepts: + - "What is TDE": tde.md + # - wal-encryption.md + - table-access-method.md - How to: - Use reference to external parameters: external-parameters.md - Decrypt an encrypted table: decrypt.md From 86b24a8987c2047f4f59bb7ea0653cba09ab1b55 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Thu, 14 Nov 2024 12:25:10 +0100 Subject: [PATCH 2/5] Added tde_heap workflow --- documentation/docs/table-access-method.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/documentation/docs/table-access-method.md b/documentation/docs/table-access-method.md index 242b2bc5..dfad99b9 100644 --- a/documentation/docs/table-access-method.md +++ b/documentation/docs/table-access-method.md @@ -34,6 +34,12 @@ CREATE TABLE table_name ( The `tde_heap` is a custom table access method that comes with the `pg_tde` extension to provide data encryption. It is automatically created **only** for the databases where you [enabled the `pg_tde` extension](setup.md) and configured the key provider. +### How does it work? + +The `tde_heap` access method works on top of the default heap access method and is a marker to point which tables require encryption. It uses the custom storage manager TDE SMGR, which becomes active only after you installed the `pg_tde` extension. + +When a table requires encryption, every data block is encrypted before it is written to disk and decrypted after reading before it is sent to the PostgreSQL core and then to the client. The encryption is done at the storage manager level. + ## Changing the default table access method From cbff7603cf0aa492a7348b8438abd95d8aa995c7 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Mon, 18 Nov 2024 17:44:47 +0100 Subject: [PATCH 3/5] PG-1116 Reworked the custom access method section after the review --- documentation/docs/table-access-method.md | 37 ++++++++++------------- 1 file changed, 16 insertions(+), 21 deletions(-) diff --git a/documentation/docs/table-access-method.md b/documentation/docs/table-access-method.md index dfad99b9..dc1cfc47 100644 --- a/documentation/docs/table-access-method.md +++ b/documentation/docs/table-access-method.md @@ -1,53 +1,47 @@ # Table access method -A table access method is the way how PostgreSQL stores the data in a table. The default table access method is `heap`. PostgreSQL organizes data in a heap structure, meaning there is no particular order to the rows in the table. Each row is stored independently, and rows are identified by their unique row identifier (TID). +A table access method is the way how PostgreSQL stores the data in a table. The default table access method is `heap`. PostgreSQL organizes data in a heap structure, meaning there is no particular order to the rows in the table. Each row is stored independently and identified by its unique row identifier (TID). -## How does the heap access method work? +## How the `heap` access method works **Insertion**: When a new row is inserted, PostgreSQL finds a free space in the tablespace and stores the row there. **Deletion**: When a row is deleted, PostgreSQL marks the space occupied by the row as free, but the data remains until it is overwritten by a new insertion. -**Updates**: Updates are handled by deleting the old row and inserting a new row with the updated values +**Updates**: PostgreSQL handles updates by deleting the old row and inserting a new row with the updated values. ## Custom access method -You can create a custom table access method for each table and instruct PostgreSQL how to store the data for you. For example, you can tailor the table access method to better suit your specific workload or data access patterns. +Custom access methods allow you to implement and define your own way of organizing data in PostgreSQL. This is useful if the default table access method doesn't meet your needs. -To define an access method, use the `CREATE ACCESS METHOD` with the `TYPE` clause set to `table`: +Custom access methods are typically available with PostgreSQL extensions. When you install an extension and enable it in PostgreSQL, a custom access method is created. -```sql -CREATE ACCESS METHOD access_method_name TYPE table; -``` +An example of such an approach is the `tde_heap` access method. It is automatically created **only** for the databases where you [enabled the `pg_tde` extension](setup.md) and configured the key provider, enabling you to encrypt the data. -To use your access method, specify the `USING` clause for the `CREATE TABLE` command: +To use a custom access method, specify the `USING` clause for the `CREATE TABLE` command: ```sql CREATE TABLE table_name ( column1 data_type, column2 data_type, ... -) USING access_method_name; +) USING tde_heap; ``` -## `tde_heap` access method - -The `tde_heap` is a custom table access method that comes with the `pg_tde` extension to provide data encryption. It is automatically created **only** for the databases where you [enabled the `pg_tde` extension](setup.md) and configured the key provider. +### How `tde_heap` works -### How does it work? - -The `tde_heap` access method works on top of the default heap access method and is a marker to point which tables require encryption. It uses the custom storage manager TDE SMGR, which becomes active only after you installed the `pg_tde` extension. +The `tde_heap` access method works on top of the default `heap` access method and is a marker to point which tables require encryption. It uses the custom storage manager TDE SMGR, which becomes active only after you installed the `pg_tde` extension. When a table requires encryption, every data block is encrypted before it is written to disk and decrypted after reading before it is sent to the PostgreSQL core and then to the client. The encryption is done at the storage manager level. - ## Changing the default table access method -You can change the default table access method so that every table in the entire database cluster is created using the custom access method. For example, you can enable data encryption by default by defining either `tde_heap_basic` or the `tde_heap` as the default table access method. +You can change the default table access method so that every table in the entire database cluster is created using the custom access method. For example, you can enable data encryption by default by defining the `tde_heap` as the default table access method. -However, consider the following before doing so: +However, consider the following before making this change: -* This is a global setting and applies across the entire database cluster and not just a single database. We recommend setting it with caution only if you created the `pg_tde` extensions for all databases. Otherwise PostgreSQL throws an error. +* This is a global setting and applies across the entire database cluster and not just a single database. +We recommend setting it with caution because all tables and materialized views created without an explicit access method in their `CREATE` statement will default to the specified table access method. * You must create the `pg_tde` extension and configure the key provider for all databases before you modify the configuration. Otherwise PostgreSQL won't find the specified access method and will throw an error. Here's how you can set the new default table access method: @@ -56,7 +50,7 @@ Here's how you can set the new default table access method: === "via the SQL statement" - Use the `ALTER SYSTEM SET` command. This requires superuser privileges. + Use the `ALTER SYSTEM SET` command. This requires superuser or ALTER SYSTEM privileges. This example shows how to set the `tde_heap` access method. Replace it with the `tde_heap_basic` if needed. @@ -93,4 +87,5 @@ Here's how you can set the new default table access method: ```sql SELECT pg_reload_conf(); + ``` From 708dc3e92c25678ecf3391e9840a075d5c72b6f1 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Tue, 26 Nov 2024 17:26:35 +0100 Subject: [PATCH 4/5] Updated the encyption flow --- documentation/docs/table-access-method.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/documentation/docs/table-access-method.md b/documentation/docs/table-access-method.md index dc1cfc47..f6ca90d5 100644 --- a/documentation/docs/table-access-method.md +++ b/documentation/docs/table-access-method.md @@ -32,7 +32,12 @@ CREATE TABLE table_name ( The `tde_heap` access method works on top of the default `heap` access method and is a marker to point which tables require encryption. It uses the custom storage manager TDE SMGR, which becomes active only after you installed the `pg_tde` extension. -When a table requires encryption, every data block is encrypted before it is written to disk and decrypted after reading before it is sent to the PostgreSQL core and then to the client. The encryption is done at the storage manager level. +WEvery data modification operation is first sent to the Buffer Manager, which updates the buffer cache. Then, it is passed to the storage manager, which then writes it to disk. When a table requires encryption, the data is sent to the TDE storage manager, where it is encrypted before written to disk. + +Similarly, when a client queries the database, the PostgreSQL core sends the request to the Buffer Manager which checks if the requested data is already in the buffer cache. If it’s not there, the Buffer Manager requests the data from the storage manager. The TDE storage manager reads the encrypted data from disk, decrypts it and loads it to the buffer cache. The Buffer Manager sends the requested data to the PostgreSQL core and then to the client. + + +Thus, the encryption is done at the storage manager level. ## Changing the default table access method From 1a9ae26e14bc447d38824dc5efcd4623f96f7918 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Tue, 26 Nov 2024 19:16:08 +0100 Subject: [PATCH 5/5] Update documentation/docs/table-access-method.md Co-authored-by: Andrew Pogrebnoi --- documentation/docs/table-access-method.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/docs/table-access-method.md b/documentation/docs/table-access-method.md index f6ca90d5..4ed69bc6 100644 --- a/documentation/docs/table-access-method.md +++ b/documentation/docs/table-access-method.md @@ -32,7 +32,7 @@ CREATE TABLE table_name ( The `tde_heap` access method works on top of the default `heap` access method and is a marker to point which tables require encryption. It uses the custom storage manager TDE SMGR, which becomes active only after you installed the `pg_tde` extension. -WEvery data modification operation is first sent to the Buffer Manager, which updates the buffer cache. Then, it is passed to the storage manager, which then writes it to disk. When a table requires encryption, the data is sent to the TDE storage manager, where it is encrypted before written to disk. +Every data modification operation is first sent to the Buffer Manager, which updates the buffer cache. Then, it is passed to the storage manager, which then writes it to disk. When a table requires encryption, the data is sent to the TDE storage manager, where it is encrypted before written to disk. Similarly, when a client queries the database, the PostgreSQL core sends the request to the Buffer Manager which checks if the requested data is already in the buffer cache. If it’s not there, the Buffer Manager requests the data from the storage manager. The TDE storage manager reads the encrypted data from disk, decrypts it and loads it to the buffer cache. The Buffer Manager sends the requested data to the PostgreSQL core and then to the client.