HDDS-11947. [Docs] [User Guide] Impala Integration. #7584

jojochuang · 2024-12-17T03:59:59Z

What changes were proposed in this pull request?

HDDS-11947. [Docs] [User Guide] Impala Integration.

Adding the doc to the old user doc site.

Please describe your PR in detail:

Add user doc for Impala integration.
Backported from the new site: HDDS-11931. [Website v2] [Docs] [User Guide] Impala Integration. ozone-site#113

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11947

How was this patch tested?

./hadoop-ozone/dev-support/checks/docs.sh passed.

https://github.com/jojochuang/ozone/actions/runs/12364445561

Change-Id: Ic09028a519199274093186b611963ebb8bfb7879

adoroszlai

Thanks @jojochuang for documenting this.

hadoop-hdds/docs/content/integration/_index.md

hadoop-hdds/docs/content/integration/Impala.md

hadoop-hdds/docs/content/integration/_index.md

errose28

Thanks for adding this @jojochuang. I think we should try to standardize the layout of the integration pages a little more. For example the way the proposed Impala and Hive pages mention supported protocols, replication types, bucket types, etc is different.

hadoop-hdds/docs/content/integration/Impala.md

Change-Id: I30a10b188276987558e5d8879b45a48b65ea6740

jojochuang · 2024-12-20T19:00:48Z

Thanks for the in-depth review! Updated the doc and here's the screenshot:

Change-Id: I33bb688d70b10bf616dfd4eef20775591cc53e40

MikaelSmith

As an Impala committer, I think this looks good and appreciate having it added to the Ozone docs.

adoroszlai

Thanks @jojochuang for updating the patch. Few formatting suggestions based on looking at the rendered version.

hadoop-hdds/docs/content/integration/Impala.md

adoroszlai · 2025-01-08T04:44:58Z

hadoop-hdds/docs/content/integration/Impala.md

+
+Starting with version 4.2.0, Apache Impala provides full support for querying data stored in Apache Ozone. To utilize this functionality, ensure that your Ozone version is 1.4.0 or later.
+
+# Supported Access Protocols


Impala is rendered as h1, so I think these sections should be h2.

Suggested change

# Supported Access Protocols

## Supported Access Protocols

adoroszlai · 2025-01-08T04:45:05Z

hadoop-hdds/docs/content/integration/Impala.md

+
+Note: The o3fs protocol is **NOT** supported by Impala.
+
+# Supported Replication Types


Suggested change

# Supported Replication Types

## Supported Replication Types

adoroszlai · 2025-01-08T04:45:13Z

hadoop-hdds/docs/content/integration/Impala.md

+* RATIS (Replication)
+* Erasure Coding
+
+# Querying Ozone Data with Impala


Suggested change

# Querying Ozone Data with Impala

## Querying Ozone Data with Impala

adoroszlai · 2025-01-08T04:45:26Z

hadoop-hdds/docs/content/integration/Impala.md

+* Managed Tables
+* External Tables
+
+## Managed Tables


Suggested change

## Managed Tables

### Managed Tables

adoroszlai · 2025-01-08T04:46:02Z

hadoop-hdds/docs/content/integration/Impala.md

+* Note: Dropping an external table in Impala does not delete the associated data.
+
+
+# Using the S3A Protocol


Suggested change

# Using the S3A Protocol

## Using the S3A Protocol

adoroszlai · 2025-01-08T04:49:38Z

hadoop-hdds/docs/content/integration/Impala.md

+If the Hive Warehouse Directory is located in Ozone, you can execute Impala queries without any changes, treating the Ozone file system like HDFS. For example:
+
+```sql
+  CREATE DATABASE d1;


The CREATE EXTERNAL TABLE example is not indented, so I think the others can be unindented as well.

Suggested change

CREATE DATABASE d1;

CREATE DATABASE d1;

adoroszlai · 2025-01-08T04:49:45Z

hadoop-hdds/docs/content/integration/Impala.md

+```
+
+```sql
+  CREATE TABLE t1 (x INT, s STRING);


Suggested change

CREATE TABLE t1 (x INT, s STRING);

CREATE TABLE t1 (x INT, s STRING);

adoroszlai · 2025-01-08T04:49:55Z

hadoop-hdds/docs/content/integration/Impala.md

+You can create managed databases, tables, or partitions at a specific Ozone path using the `LOCATION` clause. Example:
+
+```sql
+  CREATE DATABASE d1 LOCATION 'ofs://ozone1/vol1/bucket1/d1.db';


Suggested change

CREATE DATABASE d1 LOCATION 'ofs://ozone1/vol1/bucket1/d1.db';

CREATE DATABASE d1 LOCATION 'ofs://ozone1/vol1/bucket1/d1.db';

adoroszlai · 2025-01-08T04:50:03Z

hadoop-hdds/docs/content/integration/Impala.md

+```
+
+```sql
+  CREATE TABLE t1 LOCATION 'ofs://ozone1/vol1/bucket1/table1';


Suggested change

CREATE TABLE t1 LOCATION 'ofs://ozone1/vol1/bucket1/table1';

CREATE TABLE t1 LOCATION 'ofs://ozone1/vol1/bucket1/table1';

Co-authored-by: Doroszlai, Attila <[email protected]>

Change-Id: Ia3bac190ab03fc275cde4f72d5d6f8fbf73bd1fd

jojochuang · 2025-01-08T18:06:53Z

errose28

Thanks for the updates Wei-Chiu. Format LGTM and @MikaelSmith has provided a good endorsement that the content is correct.

One thing to consider before merging, should we leave this page as draft (a front matter key indicating it will not be published to the final site) until we get more integration pages completed? I think the current layout makes it look like Impala is the only supported Ozone integration.

jojochuang · 2025-01-08T18:26:57Z

That's a good point. Hmm it doesn't really work the way it's supposed to be though. It still shows up on the side menu.

jojochuang · 2025-01-08T18:33:53Z

Ah nvm i know what's going on. Updating it.

Change-Id: I3006e8dc7bf8b0f1a55f16ccc3da8555c1d231e9

adoroszlai · 2025-01-09T12:23:12Z

One thing to consider before merging, should we leave this page as draft (a front matter key indicating it will not be published to the final site) until we get more integration pages completed? I think the current layout makes it look like Impala is the only supported Ozone integration.

With draft the page will not be rendered at all, so the content is not checked for syntax errors. Hugo provides some other options to hide content.

jojochuang mentioned this pull request Dec 17, 2024

HDDS-11948. [Docs] [User Guide] DistCP integration. #7588

Open

HDDS-11947. [Docs] [User Guide] Impala Integration.

382380d

Change-Id: Ic09028a519199274093186b611963ebb8bfb7879

jojochuang force-pushed the HDDS-11931_old branch from c9689fd to 382380d Compare December 17, 2024 19:40

adoroszlai added the documentation Improvements or additions to documentation label Dec 18, 2024

adoroszlai reviewed Dec 19, 2024

View reviewed changes

hadoop-hdds/docs/content/integration/_index.md Outdated Show resolved Hide resolved

hadoop-hdds/docs/content/integration/Impala.md Show resolved Hide resolved

hadoop-hdds/docs/content/integration/_index.md Outdated Show resolved Hide resolved

errose28 reviewed Dec 19, 2024

View reviewed changes

Update

3a42bbf

Change-Id: I30a10b188276987558e5d8879b45a48b65ea6740

Typo

f86262f

Change-Id: I33bb688d70b10bf616dfd4eef20775591cc53e40

jojochuang force-pushed the HDDS-11931_old branch from 5fb5f29 to f86262f Compare December 20, 2024 19:11

MikaelSmith approved these changes Jan 8, 2025

View reviewed changes

adoroszlai reviewed Jan 8, 2025

View reviewed changes

jojochuang and others added 2 commits January 8, 2025 09:57

Update hadoop-hdds/docs/content/integration/Impala.md

b6b6419

Co-authored-by: Doroszlai, Attila <[email protected]>

Update format and identation

5f66224

Change-Id: Ia3bac190ab03fc275cde4f72d5d6f8fbf73bd1fd

errose28 reviewed Jan 8, 2025

View reviewed changes

Hide Application Integration page.

0fa460a

Change-Id: I3006e8dc7bf8b0f1a55f16ccc3da8555c1d231e9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-11947. [Docs] [User Guide] Impala Integration. #7584

HDDS-11947. [Docs] [User Guide] Impala Integration. #7584

jojochuang commented Dec 17, 2024 •

edited by errose28

Loading

adoroszlai left a comment

errose28 left a comment

jojochuang commented Dec 20, 2024

MikaelSmith left a comment

adoroszlai left a comment

adoroszlai Jan 8, 2025

adoroszlai Jan 8, 2025

adoroszlai Jan 8, 2025

adoroszlai Jan 8, 2025

adoroszlai Jan 8, 2025

adoroszlai Jan 8, 2025

adoroszlai Jan 8, 2025

adoroszlai Jan 8, 2025

adoroszlai Jan 8, 2025

jojochuang commented Jan 8, 2025

errose28 left a comment

jojochuang commented Jan 8, 2025

jojochuang commented Jan 8, 2025

adoroszlai commented Jan 9, 2025


		Starting with version 4.2.0, Apache Impala provides full support for querying data stored in Apache Ozone. To utilize this functionality, ensure that your Ozone version is 1.4.0 or later.

		# Supported Access Protocols


		Note: The o3fs protocol is NOT supported by Impala.

		# Supported Replication Types

	# Querying Ozone Data with Impala
	## Querying Ozone Data with Impala

		* Note: Dropping an external table in Impala does not delete the associated data.


		# Using the S3A Protocol

	CREATE TABLE t1 (x INT, s STRING);
	CREATE TABLE t1 (x INT, s STRING);

	CREATE DATABASE d1 LOCATION 'ofs://ozone1/vol1/bucket1/d1.db';
	CREATE DATABASE d1 LOCATION 'ofs://ozone1/vol1/bucket1/d1.db';

	CREATE TABLE t1 LOCATION 'ofs://ozone1/vol1/bucket1/table1';
	CREATE TABLE t1 LOCATION 'ofs://ozone1/vol1/bucket1/table1';

HDDS-11947. [Docs] [User Guide] Impala Integration. #7584

Are you sure you want to change the base?

HDDS-11947. [Docs] [User Guide] Impala Integration. #7584

Conversation

jojochuang commented Dec 17, 2024 • edited by errose28 Loading

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

adoroszlai left a comment

Choose a reason for hiding this comment

errose28 left a comment

Choose a reason for hiding this comment

jojochuang commented Dec 20, 2024

MikaelSmith left a comment

Choose a reason for hiding this comment

adoroszlai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jojochuang commented Jan 8, 2025

errose28 left a comment

Choose a reason for hiding this comment

jojochuang commented Jan 8, 2025

jojochuang commented Jan 8, 2025

adoroszlai commented Jan 9, 2025

jojochuang commented Dec 17, 2024 •

edited by errose28

Loading