Skip to content

Commit

Permalink
Merge pull request #13 from frischHWC/to-v2-0
Browse files Browse the repository at this point in the history
To v2 0
  • Loading branch information
frischHWC authored Mar 27, 2024
2 parents 1734e56 + c76f81a commit e017bd8
Show file tree
Hide file tree
Showing 10 changed files with 250 additions and 402 deletions.
38 changes: 16 additions & 22 deletions README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ One script to rule them all: CDP / CDP Private Cloud / CDH / HDP !

Given some machines, this script will **setup** all pre-requisites, **Install**, **Configure** a fully **secure** cluster and **Load Data** into it.


image:images/one-script-all.gif[Example of a basic launch]


## Table of tested Deployments

[.stripes-even, cols="1,1,4,1,1,1,1,4"]
Expand All @@ -12,16 +16,7 @@ Given some machines, this script will **setup** all pre-requisites, **Install**,

| ✓
| basic
| CDH 7.1.9.3 & CM 7.11.3.3
| CENTOS 7.9
| Postgres 14
| MIT KDC
| True
|

| ✓
| basic-enc
| CDH 7.1.9.3 & CM 7.11.3.3
| CDP 7.1.9.4 & CM 7.11.3.6
| CENTOS 7.9
| Postgres 14
| MIT KDC
Expand All @@ -30,7 +25,7 @@ Given some machines, this script will **setup** all pre-requisites, **Install**,

| ✓
| full
| CDH 7.1.9.3 & CM 7.11.3.3
| CDP 7.1.9.4 & CM 7.11.3.6
| CENTOS 7.9
| Postgres 14
| MIT KDC
Expand All @@ -39,7 +34,7 @@ Given some machines, this script will **setup** all pre-requisites, **Install**,

| ✓
| all-services
| CDH 7.1.9.3 & CM 7.11.3.3
| CDP 7.1.9.4 & CM 7.11.3.6
| CENTOS 7.9
| Postgres 14
| MIT KDC
Expand All @@ -48,7 +43,7 @@ Given some machines, this script will **setup** all pre-requisites, **Install**,

| ✓
| streaming
| CDH 7.1.9.3 & CM 7.11.3.3
| CDP 7.1.9.4 & CM 7.11.3.6 & CFM 2.1.6
| CENTOS 7.9
| Postgres 14
| MIT KDC
Expand All @@ -57,17 +52,16 @@ Given some machines, this script will **setup** all pre-requisites, **Install**,

| ✓
| cdp-streaming-with-efm
| CDH 7.1.9.3 & CM 7.11.3.2
| CDP 7.1.9.4 & CM 7.11.3.6 & CFM 2.1.6 & EFM 2.1.2
| RHEL 8.8
| Postgres 14
| MIT KDC
| True
| YCLOUD
| Use --jdk-version=17 (and a RHEL >= 8)

| ✓
| cdp-observability
| CDH 7.1.9.3 & CM 7.11.3
| CDP 7.1.9.4 & CM 7.11.3.6
| CENTOS 7.9
| Postgres 14
| MIT KDC
Expand All @@ -85,7 +79,7 @@ Given some machines, this script will **setup** all pre-requisites, **Install**,

| ✓
| pvc
| CDH 7.1.9.3 & CM 7.11.3.3 & PvC 1.5.2-h3
| CDP 7.1.9.4 & CM 7.11.3.2 & PvC 1.5.3
| CENTOS 7.9
| Postgres 14
| Free IPA
Expand All @@ -94,7 +88,7 @@ Given some machines, this script will **setup** all pre-requisites, **Install**,

| ✓
| all-services-pvc
| CDH 7.1.9.3 & CM 7.11.3.3 & PvC 1.5.2-h3
| CDP 7.1.9.4 & CM 7.11.3.2 & PvC 1.5.3
| CENTOS 7.9
| Postgres 14
| Free IPA
Expand Down Expand Up @@ -140,7 +134,7 @@ Given some machines, this script will **setup** all pre-requisites, **Install**,

|
| basic
| CDH 7.1.9.3 & CM 7.11.3.3
| CDP 7.1.9.4 & CM 7.11.3.6
| CENTOS 8.8
| Postgres 14
| MIT KDC
Expand All @@ -149,7 +143,7 @@ Given some machines, this script will **setup** all pre-requisites, **Install**,

|
| full
| CDH 7.1.9.3 & CM 7.11.3.3
| CDP 7.1.9.4 & CM 7.11.3.6
| CENTOS 8.8
| Postgres 14
| MIT KDC
Expand All @@ -158,7 +152,7 @@ Given some machines, this script will **setup** all pre-requisites, **Install**,

|
| streaming
| CDH 7.1.9.3 & CM 7.11.3.3
| CDP 7.1.9.4 & CM 7.11.3.6 & CFM 2.1.6
| CENTOS 8.8
| Postgres 14
| MIT KDC
Expand All @@ -167,7 +161,7 @@ Given some machines, this script will **setup** all pre-requisites, **Install**,

|
| pvc
| CDH 7.1.9.3 & CM 7.11.3.3 & PvC 1.5.2-h3
| CDP 7.1.9.4 & CM 7.11.3.2 & PvC 1.5.3
| CENTOS 8.8
| Postgres 14
| Free IPA
Expand Down
18 changes: 18 additions & 0 deletions RELEASE_NOTES.adoc
Original file line number Diff line number Diff line change
@@ -1,5 +1,23 @@
# Releases Notes

## v2.0

### New Features:

- Revamp of script to be good looking and easy to read
- Add Gif on README
- Update to latest versions
- Support for EFM
- Support for PvC 1.5.3
- Datagen version 0.4.14
### WIP:

- Issue on stuck ansible connection when importing the cluster template => Need to relaunch the playbooks and stop stuck connection
- Issue on Postgres 14 with RHEL 8 that needs to enable the service once failed and relaunch installation
- Parameters for more network and disks settings on Cloud
- PvC on AWS (Requires some PTR Records)
## v1.2

### New Features:
Expand Down
2 changes: 2 additions & 0 deletions ansible-cdp-all-services/cluster.yml
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,8 @@ clusters:
ozone.service.id: "{{ cluster_name }}"
ozone.scm.primordial.node.id: "{{ groups.cluster_master_nodes | first | default(omit) }}"
ozone.administrators: "{{ default_admin_user }}"
OZONE_MANAGER:
ozone-conf/ozone-site.xml_role_safety_valve: <property><name>ozone.om.snapshot.load.native.lib</name><value>false</value></property>

SPARK_ON_YARN:
SERVICEWIDE:
Expand Down
1 change: 0 additions & 1 deletion ansible-cdp-streaming/cluster.yml
Original file line number Diff line number Diff line change
Expand Up @@ -271,7 +271,6 @@ clusters:
repositories:
- "{{ cdh_repo }}"
- "{{ cfm_repo }}"
- "{{ cem_repo }}"

baseCluster:
name: "{{ cluster_name }}"
Expand Down
1 change: 0 additions & 1 deletion ansible-cdp-streaming/extra_vars.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ cloudera_manager_csds:
- ${CSA_SSB_CSD}
- ${CFM_CSD}
- ${CFM_REGISTRY_CSD}
- ${CEM_CSD}
cloudera_manager_options:
# set values to null to delete (and revert to default)
custom_banner_html: ""
Expand Down
159 changes: 159 additions & 0 deletions functions.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
#!/bin/bash

# Print a spin
#
# Usage:
# spin
# ....
# end_spin
sp="/-\|"
sc=0
spin() {
printf "\b${sp:sc++:1}"
((sc==${#sp})) && sc=0
}
end_spin() {
printf "\r%s\n" "$@"
}

# Print a progress bar like that :
# Progress : [########################################] 100%
#
# Usage:
# progress_bar <progress> <max_progress>
function progress_bar {
if [ $1 -le $2 ]; then
let _progress=(${1}*100/${2}*100)/100
else
let _one=($2 - 1)
let _progress=(${_one}*100/${2}*100)/100
fi
let _done=(${_progress}*4)/10
let _left=40-$_done

_fill=$(printf "%${_done}s")
_empty=$(printf "%${_left}s")

printf "\rProgress : [${_fill// /#}${_empty// /-}] ${_progress}%%"

}

# Launch a playbook with retries, timeouts and prints its progress, and check if it succeeds or not
#
# Usage:
# launch_playbook <playbook_name> "Success Message" "Error Message" <Expected_time_to_complete (Set it to 0, to print a spin instead of a progress bar)> <Timeout> <Number of retries> <true or false if it ahs to be launched remotely>
#
function launch_playbook() {
local playbook=$1
local success_message=$2
local error_message=$3
local expected_time=$4
local command_timeout=$5
local retry=$6
local remote_launch=$7

local launch_tried=0
local launch_failed=true
local launch_timeout=false

while [ $launch_tried -le $retry ] ; do
if [ $launch_tried -gt 0 ]; then
logger info "Launching a retry because installation failed, retry is : #bold:$launch_tried out of $FREE_IPA_TRIES#end_bold possible tries"
fi

if [ "${DEBUG}" = "true" ]
then
if [ "$remote_launch" = "true" ] ; then
logger debug " Command launched on remote node: ansible-playbook -i hosts --extra-vars @environment/extra_vars.yml ${playbook}.yml ${ANSIBLE_PYTHON_3_PARAMS} "
else
logger debug " Command launched: ansible-playbook -i ${HOSTS_FILE} playbooks/${playbook}/main.yml --extra-vars \"@/tmp/${playbook}_extra_vars.yml\" ${ANSIBLE_PYTHON_3_PARAMS} "
fi
fi

if [ "$remote_launch" = "true" ] ; then
log_file="${LOG_DIR}/deployment.log"
logger info " Follow progression in: #underline:$log_file "
ssh ${NODE_USER}@${NODE_0} "cd ~/deployment/ansible-repo/ ; ansible-playbook -i hosts --extra-vars @environment/extra_vars.yml ${playbook}.yml ${ANSIBLE_PYTHON_3_PARAMS}" >> $log_file 2>&1 &
else
log_file="${LOG_DIR}/${playbook}.log"
cp playbooks/${playbook}/extra_vars.yml /tmp/${playbook}_extra_vars.yml
envsubst < /tmp/${playbook}_extra_vars.yml > /tmp/${playbook}_extra_vars.yml.tmp && mv /tmp/${playbook}_extra_vars.yml.tmp /tmp/${playbook}_extra_vars.yml
logger info " Follow progression in: #underline:$log_file "
ansible-playbook -i ${HOSTS_FILE} playbooks/${playbook}/main.yml --extra-vars "@/tmp/${playbook}_extra_vars.yml" ${ANSIBLE_PYTHON_3_PARAMS} > $log_file 2>&1 &
fi

local pid=$!
local start=$SECONDS

if [ "${DEBUG}" = "true" ]
then
logger debug "pid is: $pid"
fi

local duration=$(( SECONDS - start ))
while [ $duration -lt $command_timeout ] ; do
if [ $(ps -p $pid | wc -l) -eq 1 ] ; then
break
fi
if [ $expected_time -gt 0 ] ; then
progress_bar $duration $expected_time
else
spin
fi
sleep 0.5
duration=$(( SECONDS - start ))
done
if [ $expected_time -gt 0 ] ; then
end_spin
fi
if [ $duration -gt $command_timeout ] ; then
launch_timeout=true
fi

OUTPUT=$(tail -${ANSIBLE_LINES_NUMBER} $log_file | grep -A${ANSIBLE_LINES_NUMBER} RECAP | grep -v "failed=0" | wc -l | xargs)
if [ "${OUTPUT}" == "2" ] && [ $launch_timeout = "false" ]
then
logger success "$success_message"
logger info ""
launch_failed=false
break
else
logger warn " $error_message "
if [ "${DEBUG}" = "true" ]
then
logger debug "Will kill process with pid: $pid"
fi
kill -9 $pid 2>&1 > /dev/null
fi
launch_tried=$((launch_tried + 1))
# Not elegant but only workaround as of now for postgresql on RHEL 8
if [ "$launch_timeout" = true ] && [ "$playbook" = "create_infrastructure" ] ; then
ssh ${NODE_USER}@${NODE_0} "systemctl enable postresql-14"
fi
done
if [ "${launch_failed}" == "true" ]
then
logger error "$error_message"
logger error " See details in file: #underline:$log_file "
exit 1
fi

}
Binary file added images/one-script-all.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,7 @@
"/swagger",
"/tenants",
"/proxy",
"/policies",
"{{ user.name }}_*"
"/policies"
],
"isExcludes": false,
"isRecursive": false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,7 @@
"/controller",
"/tenants",
"/provenance",
"/policies",
"/{{ user.name }}_*"
"/policies"
],
"isExcludes": false,
"isRecursive": false
Expand Down
Loading

0 comments on commit e017bd8

Please sign in to comment.