lagoon-core-keycloak keeps crashing with Helm/K8s installation

[yohanAnush]

Hi all,
I have been trying to get Lagoon working in an EKS cluster. I am using the lagoon-core chart from the official helm repository here. I have read and followed through below documentation.

1. https://docs.lagoon.sh/installing-lagoon/requirements/
2. https://docs.lagoon.sh/installing-lagoon/efs-provisioner/
3. https://docs.lagoon.sh/installing-lagoon/install-harbor/
4. https://docs.lagoon.sh/installing-lagoon/lagoon-core/

Essentially I have a working deployments of all storage provisioners recommended, Harbor, Postgres, Elasticsearch and Kibana. And when I deploy lagoon-core Helm chart, the lagoon-core-keycloak deployment and the lagoon-core-backup-handler deployment star crashing. The error logs are as follows.

.

lagoon-core-keycloak-876fcf6cc-48bqc

Waiting until lagoon-core-keycloak-db is ready
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 172.20.39.156:3306.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
Added 'admin' to '/opt/jboss/keycloak/standalone/configuration/keycloak-add-user.json', restart server to load user
-b 0.0.0.0
=========================================================================

  Using MariaDB database

=========================================================================

05:23:42,764 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 2.0.0.Final
05:23:42,824 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.13.Final
05:23:42,832 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.4.0.Final
05:23:42,982 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 17.0.1 (WildFly Core 18.0.4.Final) starting
05:23:43,089 INFO  [org.jboss.vfs] (MSC service thread 1-2) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.wildfly.extension.elytron.SSLDefinitions (jar:file:/opt/jboss/keycloak/modules/system/layers/base/org/wildfly/extension/elytron/main/wildfly-elytron-integration-18.0.4.Final.jar!/) to method com.sun.net.ssl.internal.ssl.Provider.isFIPS()
WARNING: Please consider reporting this to the maintainers of org.wildfly.extension.elytron.SSLDefinitions
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
05:23:43,700 INFO  [org.wildfly.security] (ServerService Thread Pool -- 19) ELY00001: WildFly Elytron version 1.18.3.Final
05:23:44,399 WARN  [org.wildfly.extension.elytron] (MSC service thread 1-1) WFLYELY00023: KeyStore file '/opt/jboss/keycloak/standalone/configuration/application.keystore' does not exist. Used blank.
05:23:44,414 WARN  [org.wildfly.extension.elytron] (MSC service thread 1-1) WFLYELY01084: KeyStore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self-signed certificate for host localhost
05:23:44,431 INFO  [org.jboss.as.patching] (MSC service thread 1-1) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
05:23:44,617 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
05:23:44,619 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 17.0.1 (WildFly Core 18.0.4.Final) started in 1848ms - Started 49 of 72 services (24 services are lazy, passive or on-demand)
The batch executed successfully
05:23:44,769 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: Keycloak 17.0.1 (WildFly Core 18.0.4.Final) stopped in 15ms
05:23:45,917 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 2.0.0.Final
05:23:45,978 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.13.Final
05:23:45,985 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.4.0.Final
05:23:46,086 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 17.0.1 (WildFly Core 18.0.4.Final) starting
05:23:46,166 INFO  [org.jboss.vfs] (MSC service thread 1-2) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.wildfly.extension.elytron.SSLDefinitions (jar:file:/opt/jboss/keycloak/modules/system/layers/base/org/wildfly/extension/elytron/main/wildfly-elytron-integration-18.0.4.Final.jar!/) to method com.sun.net.ssl.internal.ssl.Provider.isFIPS()
WARNING: Please consider reporting this to the maintainers of org.wildfly.extension.elytron.SSLDefinitions
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
05:23:46,825 INFO  [org.wildfly.security] (ServerService Thread Pool -- 21) ELY00001: WildFly Elytron version 1.18.3.Final
05:23:47,661 WARN  [org.wildfly.extension.elytron] (MSC service thread 1-1) WFLYELY00023: KeyStore file '/opt/jboss/keycloak/standalone/configuration/application.keystore' does not exist. Used blank.
05:23:47,668 WARN  [org.wildfly.extension.elytron] (MSC service thread 1-2) WFLYELY01084: KeyStore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self-signed certificate for host localhost
05:23:47,690 INFO  [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
05:23:47,800 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
05:23:47,803 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 17.0.1 (WildFly Core 18.0.4.Final) started in 1879ms - Started 49 of 79 services (31 services are lazy, passive or on-demand)
The batch executed successfully
05:23:47,941 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: Keycloak 17.0.1 (WildFly Core 18.0.4.Final) stopped in 13ms
Executing: /opt/jboss/startup-scripts/00-configure-lagoon.sh
Executing: /opt/jboss/startup-scripts/10-newrelic.sh
Keycloak still not running, waiting 5 seconds
Executing cli script: /opt/jboss/startup-scripts/standalone-configure-ds-pool.cli
05:23:49,369 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 2.0.0.Final
05:23:49,453 INFO  [org.jboss.msc] (CLI command executor) JBoss MSC version 1.4.13.Final
05:23:49,462 INFO  [org.jboss.threads] (CLI command executor) JBoss Threads version 2.4.0.Final
05:23:49,563 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: Keycloak 17.0.1 (WildFly Core 18.0.4.Final) starting
05:23:49,623 INFO  [org.jboss.vfs] (MSC service thread 1-1) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.wildfly.extension.elytron.SSLDefinitions (jar:file:/opt/jboss/keycloak/modules/system/layers/base/org/wildfly/extension/elytron/main/wildfly-elytron-integration-18.0.4.Final.jar!/) to method com.sun.net.ssl.internal.ssl.Provider.isFIPS()
WARNING: Please consider reporting this to the maintainers of org.wildfly.extension.elytron.SSLDefinitions
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
05:23:50,192 INFO  [org.wildfly.security] (ServerService Thread Pool -- 21) ELY00001: WildFly Elytron version 1.18.3.Final
05:23:50,960 WARN  [org.wildfly.extension.elytron] (MSC service thread 1-1) WFLYELY00023: KeyStore file '/opt/jboss/keycloak/standalone/configuration/application.keystore' does not exist. Used blank.
05:23:50,972 WARN  [org.wildfly.extension.elytron] (MSC service thread 1-1) WFLYELY01084: KeyStore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self-signed certificate for host localhost
05:23:50,984 INFO  [org.jboss.as.patching] (MSC service thread 1-1) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
05:23:51,103 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
05:23:51,106 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 17.0.1 (WildFly Core 18.0.4.Final) started in 1731ms - Started 49 of 79 services (31 services are lazy, passive or on-demand)
The batch executed successfully
05:23:51,227 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: Keycloak 17.0.1 (WildFly Core 18.0.4.Final) stopped in 14ms
05:23:51,234 INFO  [org.jboss.modules] (CLI command executor) JBoss Modules version 2.0.0.Final
05:23:51,238 INFO  [org.jboss.as] (MSC service thread 2-1) WFLYSRV0049: Keycloak 17.0.1 (WildFly Core 18.0.4.Final) starting
05:23:51,245 INFO  [org.jboss.vfs] (MSC service thread 2-1) VFS000002: Failed to clean existing content for temp file provider of type temp. Enable DEBUG level log to find what caused this
05:23:51,351 INFO  [org.jboss.as.patching] (MSC service thread 2-1) WFLYPAT0050: Keycloak cumulative patch ID is: base, one-off patches include: none
05:23:51,351 WARN  [org.wildfly.extension.elytron] (MSC service thread 2-2) WFLYELY00023: KeyStore file '/opt/jboss/keycloak/standalone/configuration/application.keystore' does not exist. Used blank.
05:23:51,353 WARN  [org.wildfly.extension.elytron] (MSC service thread 2-2) WFLYELY01084: KeyStore /opt/jboss/keycloak/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self-signed certificate for host localhost
05:23:51,369 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
05:23:51,370 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Keycloak 17.0.1 (WildFly Core 18.0.4.Final) started in 136ms - Started 49 of 72 services (24 services are lazy, passive or on-demand)
The batch executed successfully
05:23:51,450 INFO  [org.jboss.as] (MSC service thread 2-1) WFLYSRV0050: Keycloak 17.0.1 (WildFly Core 18.0.4.Final) stopped in 11ms
JAVA_OPTS already set in environment; overriding default settings with values: -server -Xms512m -Xmx4096m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true

.....

05:23:59,716 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0002: Started realms cache from keycloak container
05:23:59,778 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0273: Excluded subsystem webservices via jboss-deployment-structure.xml does not exist.
05:23:59,932 INFO  [org.keycloak.services] (ServerService Thread Pool -- 65) KC-SERVICES0001: Loading config from standalone.xml or domain.xml
05:23:59,999 INFO  [org.keycloak.common.Profile] (ServerService Thread Pool -- 65) Preview feature enabled: admin_fine_grained_authz
05:24:00,000 INFO  [org.keycloak.common.Profile] (ServerService Thread Pool -- 65) Preview feature enabled: token_exchange
05:24:00,000 WARN  [org.keycloak.common.Profile] (ServerService Thread Pool -- 65) Deprecated feature enabled: upload_scripts
05:24:00,000 WARN  [org.keycloak.common.Profile] (ServerService Thread Pool -- 65) Preview feature enabled: scripts
05:24:00,477 INFO  [org.keycloak.url.DefaultHostnameProviderFactory] (ServerService Thread Pool -- 65) Frontend: http://lagoon-core-keycloak:8080/auth, Admin: <frontend>, Backend: <request>
05:24:00,607 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started realmRevisions cache from keycloak container
05:24:00,615 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started userRevisions cache from keycloak container
05:24:00,623 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0002: Started authorizationRevisions cache from keycloak container
05:24:00,626 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (ServerService Thread Pool -- 65) Node name: lagoon-core-keycloak-876fcf6cc-48bqc, Site name: null
05:24:00,716 WARN  [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 65) IJ000604: Throwable while attempting to get a new connection: null: javax.resource.ResourceException: IJ031084: Unable to create connection
	at [email protected]//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:364)
	at [email protected]//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:371)
	at [email protected]//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:287)
	at [email protected]//org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.createConnectionEventListener(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1328)
	at [email protected]//org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:505)
	at [email protected]//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:640)
	at [email protected]//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:605)
	at [email protected]//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:624)
	at [email protected]//org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:440)
	at [email protected]//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789)
	at [email protected]//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151)
	at [email protected]//org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64)
	at [email protected]//org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.getConnection(DefaultJpaConnectionProviderFactory.java:414)
	at [email protected]//org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider.lazyInit(LiquibaseDBLockProvider.java:65)
	at [email protected]//org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider.lambda$waitForLock$2(LiquibaseDBLockProvider.java:96)
	at [email protected]//org.keycloak.models.utils.KeycloakModelUtils.suspendJtaTransaction(KeycloakModelUtils.java:753)
	at [email protected]//org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider.waitForLock(LiquibaseDBLockProvider.java:94)
	at [email protected]//org.keycloak.services.resources.KeycloakApplication$1.run(KeycloakApplication.java:134)
	at [email protected]//org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:239)
	at [email protected]//org.keycloak.services.resources.KeycloakApplication.startup(KeycloakApplication.java:128)
	at [email protected]//org.keycloak.provider.wildfly.WildflyPlatform.onStartup(WildflyPlatform.java:36)
	at [email protected]//org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:114)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at [email protected]//org.jboss.resteasy.core.ConstructorInjectorImpl.constructOutsideRequest(ConstructorInjectorImpl.java:225)
	at [email protected]//org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:209)
	at [email protected]//org.jboss.resteasy.core.providerfactory.Utils.createProviderInstance(Utils.java:102)
	at [email protected]//org.jboss.resteasy.core.providerfactory.ResteasyProviderFactoryImpl.createProviderInstance(ResteasyProviderFactoryImpl.java:1385)
	at [email protected]//org.jboss.resteasy.core.ResteasyDeploymentImpl.createApplication(ResteasyDeploymentImpl.java:418)
	at [email protected]//org.jboss.resteasy.core.ResteasyDeploymentImpl.initializeObjects(ResteasyDeploymentImpl.java:265)
	at [email protected]//org.jboss.resteasy.core.ResteasyDeploymentImpl.startInternal(ResteasyDeploymentImpl.java:137)
	at [email protected]//org.jboss.resteasy.core.ResteasyDeploymentImpl.start(ResteasyDeploymentImpl.java:121)
	at [email protected]//org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:144)
	at [email protected]//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:42)
	at [email protected]//io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
	at org.wildfly.security.elytron-web.undertow-server-servlet@1.10.1.Final//org.wildfly.elytron.web.undertow.server.servlet.RunAsLifecycleInterceptor.doIt(RunAsLifecycleInterceptor.java:70)
	at org.wildfly.security.elytron-web.undertow-server-servlet@1.10.1.Final//org.wildfly.elytron.web.undertow.server.servlet.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:76)
	at [email protected]//io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
	at [email protected]//io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:309)
	at [email protected]//io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:145)
	at [email protected]//io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:588)
	at [email protected]//io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:559)
	at [email protected]//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
	at [email protected]//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
	at [email protected]//io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:601)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:106)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:87)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at [email protected]//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at [email protected]//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
	at java.base/java.lang.Thread.run(Thread.java:829)
	at [email protected]//org.jboss.threads.JBossThread.run(JBossThread.java:513)
Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not connect to address=(host=lagoon-core-keycloak-db)(port=3306)(type=master) : Access denied for user 'keycloak'@'10.20.45.159' (using password: YES)
Current charset is UTF-8. If password has been set using other charset, consider using option 'passwordCharacterEncoding'
	at org.mariadb.jdbc//org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:239)
	at org.mariadb.jdbc//org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1243)
	at org.mariadb.jdbc//org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:622)
	at org.mariadb.jdbc//org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:142)
	at org.mariadb.jdbc//org.mariadb.jdbc.Driver.connect(Driver.java:86)
	at [email protected]//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:335)
	... 60 more
Caused by: java.sql.SQLException: Access denied for user 'keycloak'@'10.20.45.159' (using password: YES)
Current charset is UTF-8. If password has been set using other charset, consider using option 'passwordCharacterEncoding'
	at org.mariadb.jdbc//org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:726)
	at org.mariadb.jdbc//org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:507)
	at org.mariadb.jdbc//org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1238)
	... 64 more

05:24:00,725 FATAL [org.keycloak.services] (ServerService Thread Pool -- 65) Error during startup: java.lang.RuntimeException: Failed to connect to database
	at [email protected]//org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.getConnection(DefaultJpaConnectionProviderFactory.java:420)
	at [email protected]//org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider.lazyInit(LiquibaseDBLockProvider.java:65)
	at [email protected]//org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider.lambda$waitForLock$2(LiquibaseDBLockProvider.java:96)
	at [email protected]//org.keycloak.models.utils.KeycloakModelUtils.suspendJtaTransaction(KeycloakModelUtils.java:753)
	at [email protected]//org.keycloak.connections.jpa.updater.liquibase.lock.LiquibaseDBLockProvider.waitForLock(LiquibaseDBLockProvider.java:94)
	at [email protected]//org.keycloak.services.resources.KeycloakApplication$1.run(KeycloakApplication.java:134)
	at [email protected]//org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:239)
	at [email protected]//org.keycloak.services.resources.KeycloakApplication.startup(KeycloakApplication.java:128)
	at [email protected]//org.keycloak.provider.wildfly.WildflyPlatform.onStartup(WildflyPlatform.java:36)
	at [email protected]//org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:114)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at [email protected]//org.jboss.resteasy.core.ConstructorInjectorImpl.constructOutsideRequest(ConstructorInjectorImpl.java:225)
	at [email protected]//org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:209)
	at [email protected]//org.jboss.resteasy.core.providerfactory.Utils.createProviderInstance(Utils.java:102)
	at [email protected]//org.jboss.resteasy.core.providerfactory.ResteasyProviderFactoryImpl.createProviderInstance(ResteasyProviderFactoryImpl.java:1385)
	at [email protected]//org.jboss.resteasy.core.ResteasyDeploymentImpl.createApplication(ResteasyDeploymentImpl.java:418)
	at [email protected]//org.jboss.resteasy.core.ResteasyDeploymentImpl.initializeObjects(ResteasyDeploymentImpl.java:265)
	at [email protected]//org.jboss.resteasy.core.ResteasyDeploymentImpl.startInternal(ResteasyDeploymentImpl.java:137)
	at [email protected]//org.jboss.resteasy.core.ResteasyDeploymentImpl.start(ResteasyDeploymentImpl.java:121)
	at [email protected]//org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:144)
	at [email protected]//org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:42)
	at [email protected]//io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
	at org.wildfly.security.elytron-web.undertow-server-servlet@1.10.1.Final//org.wildfly.elytron.web.undertow.server.servlet.RunAsLifecycleInterceptor.doIt(RunAsLifecycleInterceptor.java:70)
	at org.wildfly.security.elytron-web.undertow-server-servlet@1.10.1.Final//org.wildfly.elytron.web.undertow.server.servlet.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:76)
	at [email protected]//io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
	at [email protected]//io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:309)
	at [email protected]//io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:145)
	at [email protected]//io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:588)
	at [email protected]//io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:559)
	at [email protected]//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
	at [email protected]//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1544)
	at [email protected]//io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:601)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:106)
	at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:87)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at [email protected]//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at [email protected]//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
	at java.base/java.lang.Thread.run(Thread.java:829)
	at [email protected]//org.jboss.threads.JBossThread.run(JBossThread.java:513)
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/KeycloakDS
	at [email protected]//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:159)
	at [email protected]//org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64)
	at [email protected]//org.keycloak.connections.jpa.DefaultJpaConnectionProviderFactory.getConnection(DefaultJpaConnectionProviderFactory.java:414)
	... 48 more
Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/KeycloakDS
	at [email protected]//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:690)
	at [email protected]//org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:440)
	at [email protected]//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789)
	at [email protected]//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151)
	... 50 more
Caused by: javax.resource.ResourceException: IJ031084: Unable to create connection
	at [email protected]//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:364)
	at [email protected]//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:371)
	at [email protected]//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:287)
	at [email protected]//org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.createConnectionEventListener(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1328)
	at [email protected]//org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:505)
	at [email protected]//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:640)
	at [email protected]//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:605)
	at [email protected]//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:624)
	... 53 more
Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not connect to address=(host=lagoon-core-keycloak-db)(port=3306)(type=master) : Access denied for user 'keycloak'@'10.20.45.159' (using password: YES)
Current charset is UTF-8. If password has been set using other charset, consider using option 'passwordCharacterEncoding'
	at org.mariadb.jdbc//org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:239)
	at org.mariadb.jdbc//org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1243)
	at org.mariadb.jdbc//org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:622)
	at org.mariadb.jdbc//org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:142)
	at org.mariadb.jdbc//org.mariadb.jdbc.Driver.connect(Driver.java:86)
	at [email protected]//org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:335)
	... 60 more
Caused by: java.sql.SQLException: Access denied for user 'keycloak'@'10.20.45.159' (using password: YES)
Current charset is UTF-8. If password has been set using other charset, consider using option 'passwordCharacterEncoding'
	at org.mariadb.jdbc//org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:726)
	at org.mariadb.jdbc//org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:507)
	at org.mariadb.jdbc//org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1238)
	... 64 more

.

lagoon-core-backup-handler-857c77cdcc-pfxhz

lagoon-core-broker (172.20.123.199:5672) open
2024/03/15 05:24:59 rabbit failure, error is Exception (403) Reason: "username or password not allowed":
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x69b077]

goroutine 1 [running]:
github.com/isayme/go-amqp-reconnect/rabbitmq.(*Connection).Channel(0x0)
	/go/pkg/mod/github.com/isayme/[email protected]/rabbitmq/rabbitmq.go:20 +0x17
github.com/uselagoon/lagoon/services/backup-handler/internal/handler.(*BackupHandler).initAmqp(0xc0001342c0)
	/go/src/github.com/uselagoon/lagoon/services/backup-handler/internal/handler/main.go:70 +0x72
github.com/uselagoon/lagoon/services/backup-handler/internal/handler.NewBackupHandler({{0xc00002cb81, 0x12}, {0xc00003003e, 0x4}, {0xc0000300d2, 0x6}, {0xc00002a512, 0x20}, {0x740c6e, 0x15}, ...}, ...)
	/go/src/github.com/uselagoon/lagoon/services/backup-handler/internal/handler/main.go:60 +0x2b7
main.main()
	/go/src/github.com/uselagoon/lagoon/services/backup-handler/main.go:65 +0x4d4

.
Below is the list of all pods running as a part of the Lagoon stack.

NAME                                              READY   STATUS             RESTARTS        AGE
harbor-core-569c7d6574-vksgj                      1/1     Running            0               23h
harbor-database-0                                 1/1     Running            0               23h
harbor-jobservice-5bfc4d5f9b-9wfj5                1/1     Running            2 (9m45s ago)   23h
harbor-nginx-784977c948-qn8m5                     1/1     Running            0               23h
harbor-portal-6b8c795cf-bj8ng                     1/1     Running            0               23h
harbor-redis-0                                    1/1     Running            0               23h
harbor-registry-7f95dccf85-xjz7h                  2/2     Running            0               23h
harbor-trivy-0                                    1/1     Running            0               23h
lagoon-core-actions-handler-669787cc7-rzb6j       1/1     Running            1 (4m2s ago)    23h
lagoon-core-actions-handler-669787cc7-tq9kw       1/1     Running            1 (4m2s ago)    23h
lagoon-core-api-5b9c8ddf7-ltmgk                   0/1     Running            1 (28s ago)     23h
lagoon-core-api-5b9c8ddf7-z2qrp                   0/1     Running            1 (33s ago)     23h
lagoon-core-api-db-0                              1/1     Running            0               23h
lagoon-core-api-redis-6bfdf87cb5-6m6dz            1/1     Running            0               23h
lagoon-core-auth-server-7c8685d4d8-bp6m2          1/1     Running            0               23h
lagoon-core-auth-server-7c8685d4d8-d8249          1/1     Running            0               23h
lagoon-core-backup-handler-857c77cdcc-pfxhz       0/1     CrashLoopBackOff   7 (3m23s ago)   23h
lagoon-core-backup-handler-857c77cdcc-vd5rd       0/1     CrashLoopBackOff   7 (3m38s ago)   23h
lagoon-core-broker-0                              1/1     Running            0               23h
lagoon-core-broker-1                              1/1     Running            0               9m32s
lagoon-core-broker-2                              1/1     Running            0               9m3s
lagoon-core-drush-alias-7f4fcc8954-cs5d2          1/1     Running            0               23h
lagoon-core-drush-alias-7f4fcc8954-r4jzp          1/1     Running            0               23h
lagoon-core-keycloak-876fcf6cc-48bqc              0/1     CrashLoopBackOff   6 (75s ago)     23h
lagoon-core-keycloak-db-0                         1/1     Running            0               23h
lagoon-core-logs2notifications-77494c5697-hd82s   1/1     Running            1 (5m27s ago)   23h
lagoon-core-logs2notifications-77494c5697-nhrz6   1/1     Running            1 (5m25s ago)   23h
lagoon-core-ssh-77b69d845d-9jhlz                  1/1     Running            0               23h
lagoon-core-ssh-77b69d845d-wb84m                  1/1     Running            0               23h
lagoon-core-ui-7bf99f7-mxzzp                      1/1     Running            0               23h
lagoon-core-ui-7bf99f7-qcbp8                      1/1     Running            0               23h
lagoon-core-webhook-handler-64f74785f4-6bhlk      1/1     Running            0               23h
lagoon-core-webhook-handler-64f74785f4-dcbkf      1/1     Running            0               23h
lagoon-core-webhooks2tasks-778dd86c59-b4kwz       1/1     Running            0               23h
lagoon-core-webhooks2tasks-778dd86c59-vqcwj       1/1     Running            0               23h
lagoon-core-workflows-5d9b48bf48-b9ft8            1/1     Running            1 (5m25s ago)   23h
lagoon-core-workflows-5d9b48bf48-k6wcz            1/1     Running            1 (5m23s ago)   23h

Thank you for taking a look at, and any tip/suggestion is appreciated.

Regards,
Anushka.

[tobybellwood]

it looks like a password mismatch

Caused by: java.sql.SQLInvalidAuthorizationSpecException: Could not connect to address=(host=lagoon-core-keycloak-db)(port=3306)(type=master) : Access denied for user 'keycloak'@'10.20.45.159' (using password: YES)

If you check the following env vars in kubernetes:

in lagoon-core-keycloak-db: MARIADB_PASSWORD
in lagoon-core-keycloak: DB_PASSWORD

They should both match (they come from the lagoon-core-keycloak secret DB_PASSWORD value.

Likewise the lagoon-core-broker secret is what's used to connect the backup-handler.

It seems like your secrets may be out of whack slightly?

[yohanAnush]

Hey @tobybellwood, thank you! You put me on the right path cuz I ended up finding another such instance of DB passwords not matching (with Core API for api user) and corrected it as well. Now I can see that all the pods are able to talk to their respective DB instances. However, the core-api pod would still not become ready and from what I can in the logs, it looks like it's related to NewRelic license key? I'm using uselagoon/api:v2.17.0 image.

.

lagoon-core-api-789dc49cbb-96n4j

[2024-03-15 09:15:35]: Waiting until lagoon-core-api-db is ready...
[2024-03-15 09:15:39] [info]: Waiting until keycloak is ready...
New Relic for Node.js halted startup due to an error:
Error: Not starting without license key!
    at onNextTick (/app/node_modules/newrelic/lib/agent.js:226:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
(node:7) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.

Please migrate your code to use AWS SDK for JavaScript (v3).
For more information, check the migration guide at https://a.co/7PzMCcy
(Use `node --trace-warnings ...` to show where the warning was created)
[2024-03-15 09:15:39] [error]: lagoon-logs: Not connected, error: undefined
[2024-03-15 09:15:39] [error]: lagoon-actions: Not connected, error: undefined
[2024-03-15 09:15:39] [error]: lagoon-tasks: Not connected, error: undefined
[2024-03-15 09:15:44] [error]: lagoon-logs: Not connected, error: undefined

[tobybellwood]

The new relic isn't needed to startup - does the ui startup?

[yohanAnush]

@tobybellwood, I see; yes the UI does start up and takes me to the keycloak login page. However when I try the default admin credentials it says login failed and produces the below log in keycloak pod.

04:48:45,213 WARN  [org.keycloak.events] (default task-13) type=LOGIN_ERROR, realmId=master, clientId=admin-cli, userId=64b45e65-a61a-459b-b69e-8b6a37163c2d, ipAddress=10.20.39.69, error=invalid_user_credentials, auth_method=openid-connect, grant_type=password, client_auth_method=client-secret, username=admin, authSessionParentId=99ca365c-a378-405f-8379-771ec66f74ac, authSessionTabId=qTYzDXsEHmA

I used admin as the username and then queried the lagoon-core-keycloak secret by running kubectl -n lagoon-core get secret lagoon-core-keycloak -o jsonpath="{.data.KEYCLOAK_ADMIN_PASSWORD}" | base64 --decode as mentioned here. Since the autogenerated password didn't work I also tried manually setting keycloakAdminPassword attribute in values.yaml file and that too does not work and produces the same error log. Is there anyway I can manually set the password of the admin user for keycloak through keycloak DB?