Drop Hadoop 3.1 3.2 runtime support

[jojochuang]

I'd like to get your feedback on dropping Hadoop 3.1 3.2 runtime support.

Currently, Ozone is built with Hadoop 3.3.6 dependency. Our test matrix certifies Ozone execution with Hadoop 2.7, 3.1, 3.2 and 3.3.6. I am not familiar with the history of this, but given where we are, how do we feel about dropping the test matrix against Hadoop 3.1, 3.2 and 3.3.6?

Also Hadoop 2.7 is now at the end of life. If there are still Hadoop 2 users, can we bump to Hadoop 2.10.1 which is the last Hadoop 2 release?

I am asking because more and more Ozone features depend on newer Hadoop releases. We have to workaround older releases by various hacks and it's not scalable. As an example, to support unbuffer and bytebuffer-read, we added CapableOzoneFSInputStream that is not in ozonefs-hadoop2. A recent example is HDDS-8436 where I had to copy several Hadoop classes into Ozone code.

Looking forward to hearing from you,
Weichiu

[kerneltime]

Dropping Hadoop 2 support can simplify the build as well.

[arp7]

I am strongly in favor of dropping Hadoop 2 support, it will simplify our build and support matrix significantly. However it will be good to have diverse opinions from the Ozone community before we make such an impactful change.

[michelsumbul]

Hello,

I think it make sense to fully drop Hadoop 2.x
Regarding Hadoop 3.1 and 3.2, we still have a production cluster on Hadoop 3.1.2. If ozone is not tested with 3.1 anymore we could have some surprise when moving from Hadoop to Ozone. For example we will not be able to have a spark app with 3.1 and 3.3.6 at the same time.

Also it will be good to still support 3.3.5 as its the version used by the Trino community. Knowing how big the Trino community is it will be good to support it.