Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kubernetes Compatibility #527

Open
pamanseau opened this issue Oct 9, 2024 · 5 comments
Open

Kubernetes Compatibility #527

pamanseau opened this issue Oct 9, 2024 · 5 comments

Comments

@pamanseau
Copy link

When deploying in Kubernetes the application indexer-coordinator is not connecting to Postgres.

With NODE_ENV=local, the connection works, but then it tries to connect to the docker.sock and terminates.
With NODE_ENV=production the error Unable to connect to the database

Tested with either the ENV variables or command arguments

DB_HOST: graph.postgres
DB_NAME: coordinator
DB_USERNAME: postgres
DB_PASSWORD: xxx-xxx-xxx-xxx
DB_PORT: 5432
LOG_LEVEL: debug
NODE_ENV: production
          command: [ "node", "dist/main.js" ]
          args:
            - --network-endpoint=https://mainnet.base.org
            - --network=mainnet
            - --postgres-host=graph.postgres
            - --postgres-database=coordinator
            - --postgres-username=postgres
            - --postgres-password=xxx-xxx-xxx-xxx
            - --postgres-port=5432
            - --postgres-ssl-mode=disabled
            - --port=8000
            - --start-port=3100
            - --secret-key=287sjsd928aD2
            - --ipfs=https://unauthipfs.subquery.network/ipfs/api/v0
@pamanseau pamanseau changed the title ERROR Unable to connect to the database in Kubernetes Unable to connect to the database in Kubernetes Oct 9, 2024
@pamanseau pamanseau mentioned this issue Oct 9, 2024
Open
@pamanseau
Copy link
Author

pamanseau commented Oct 10, 2024 •
edited
Loading

NODE_ENV=production

graphql-tag enforces all fragment names across your application to be unique; read more about
this in the docs: http://dev.apollodata.com/core/fragments.html#unique-names
2024-10-09T11:11:11.145Z <nestjs> INFO Starting Nest application...
2024-10-09T11:11:11.200Z <nestjs> INFO TypeOrmModule dependencies initialized
2024-10-09T11:11:11.200Z <nestjs> INFO SubscriptionModule dependencies initialized
2024-10-09T11:11:11.200Z <nestjs> INFO DBModule dependencies initialized
2024-10-09T11:11:11.200Z <nestjs> INFO ConfigureModule dependencies initialized
2024-10-09T11:11:11.200Z <nestjs> INFO TypeOrmModule dependencies initialized
2024-10-09T11:11:11.202Z <nestjs> INFO PrometheusModule dependencies initialized
2024-10-09T11:11:11.202Z <nestjs> INFO DiscoveryModule dependencies initialized
2024-10-09T11:11:11.202Z <nestjs> INFO ServeStaticModule dependencies initialized
2024-10-09T11:11:11.202Z <nestjs> INFO NetworkModule dependencies initialized
2024-10-09T11:11:11.202Z <nestjs> INFO MetricsModule dependencies initialized
2024-10-09T11:11:11.202Z <nestjs> INFO ScheduleModule dependencies initialized
2024-10-09T11:11:11.202Z <nestjs> INFO EventEmitterModule dependencies initialized
2024-10-09T11:11:11.203Z <nestjs> INFO GraphQLSchemaBuilderModule dependencies initialized
2024-10-09T11:11:11.203Z <nestjs> INFO GraphQLModule dependencies initialized
2024-10-09T11:11:11.553Z <nestjs> ERROR Unable to connect to the database. Retrying (1)...
2024-10-09T11:11:14.710Z <nestjs> ERROR Unable to connect to the database. Retrying (2)...
2024-10-09T11:11:17.904Z <nestjs> ERROR Unable to connect to the database. Retrying (3)...
2024-10-09T11:11:21.083Z <nestjs> ERROR Unable to connect to the database. Retrying (4)...
2024-10-09T11:11:24.242Z <nestjs> ERROR Unable to connect to the database. Retrying (5)...
2024-10-09T11:11:27.507Z <nestjs> ERROR Unable to connect to the database. Retrying (6)...
2024-10-09T11:11:30.710Z <nestjs> ERROR Unable to connect to the database. Retrying (7)...
2024-10-09T11:11:33.902Z <nestjs> ERROR Unable to connect to the database. Retrying (8)...
2024-10-09T11:11:37.093Z <nestjs> ERROR Unable to connect to the database. Retrying (9)...
2024-10-09T11:11:37.093Z <nestjs> ERROR function uuid_generate_v4() does not exist

NODE_ENV=local

Warning: fragment with name ProjectFields already exists.
graphql-tag enforces all fragment names across your application to be unique; read more about
this in the docs: http://dev.apollodata.com/core/fragments.html#unique-names
2024-10-09T11:21:09.428Z <[95mnestjs[39m> [32mINFO[39m Starting Nest application... 
2024-10-09T11:21:09.485Z <[95mnestjs[39m> [32mINFO[39m TypeOrmModule dependencies initialized 
2024-10-09T11:21:09.485Z <[95mnestjs[39m> [32mINFO[39m SubscriptionModule dependencies initialized 
2024-10-09T11:21:09.485Z <[95mnestjs[39m> [32mINFO[39m DBModule dependencies initialized 
2024-10-09T11:21:09.485Z <[95mnestjs[39m> [32mINFO[39m ConfigureModule dependencies initialized 
2024-10-09T11:21:09.485Z <[95mnestjs[39m> [32mINFO[39m TypeOrmModule dependencies initialized 
2024-10-09T11:21:09.486Z <[95mnestjs[39m> [32mINFO[39m PrometheusModule dependencies initialized 
2024-10-09T11:21:09.486Z <[95mnestjs[39m> [32mINFO[39m DiscoveryModule dependencies initialized 
2024-10-09T11:21:09.487Z <[95mnestjs[39m> [32mINFO[39m ServeStaticModule dependencies initialized 
2024-10-09T11:21:09.487Z <[95mnestjs[39m> [32mINFO[39m NetworkModule dependencies initialized 
2024-10-09T11:21:09.487Z <[95mnestjs[39m> [32mINFO[39m MetricsModule dependencies initialized 
2024-10-09T11:21:09.487Z <[95mnestjs[39m> [32mINFO[39m ScheduleModule dependencies initialized 
2024-10-09T11:21:09.487Z <[95mnestjs[39m> [32mINFO[39m EventEmitterModule dependencies initialized 
2024-10-09T11:21:09.487Z <[95mnestjs[39m> [32mINFO[39m GraphQLSchemaBuilderModule dependencies initialized 
2024-10-09T11:21:09.487Z <[95mnestjs[39m> [32mINFO[39m GraphQLModule dependencies initialized 
query: SELECT * FROM current_schema()
query: CREATE EXTENSION IF NOT EXISTS "uuid-ossp"
query: SELECT version();
query: SELECT * FROM "information_schema"."tables" WHERE "table_schema" = 'public' AND "table_name" = 'migrations'
query: SELECT * FROM "migrations" "migrations" ORDER BY "id" DESC
2024-10-09T11:21:09.725Z <[95mnestjs[39m> [32mINFO[39m TypeOrmCoreModule dependencies initialized 
2024-10-09T11:21:09.726Z <[95mnestjs[39m> [32mINFO[39m TypeOrmModule dependencies initialized 
2024-10-09T11:21:09.726Z <[95mnestjs[39m> [32mINFO[39m TypeOrmModule dependencies initialized 
2024-10-09T11:21:09.726Z <[95mnestjs[39m> [32mINFO[39m TypeOrmModule dependencies initialized 
2024-10-09T11:21:09.726Z <[95mnestjs[39m> [32mINFO[39m TypeOrmModule dependencies initialized 
2024-10-09T11:21:09.726Z <[95mnestjs[39m> [32mINFO[39m TypeOrmModule dependencies initialized 
2024-10-09T11:21:09.726Z <[95mnestjs[39m> [32mINFO[39m TypeOrmModule dependencies initialized 
2024-10-09T11:21:09.726Z <[95mnestjs[39m> [32mINFO[39m DBModule dependencies initialized 
2024-10-09T11:21:09.790Z <[95mnestjs[39m> [32mINFO[39m ChainModule dependencies initialized 
2024-10-09T11:21:09.791Z <[95mnestjs[39m> [32mINFO[39m ConfigModule dependencies initialized 
2024-10-09T11:21:09.791Z <[95mnestjs[39m> [32mINFO[39m StatsModule dependencies initialized 
2024-10-09T11:21:09.791Z <[95mnestjs[39m> [32mINFO[39m CoreModule dependencies initialized 
2024-10-09T11:21:09.791Z <[95mnestjs[39m> [32mINFO[39m AppModule dependencies initialized 
2024-10-09T11:21:09.795Z <[95mmonitor[39m> [32mINFO[39m check node health started 
2024-10-09T11:21:09.795Z <[95mnestjs[39m> [32mINFO[39m PaygModule dependencies initialized 
2024-10-09T11:21:09.795Z <[95mnestjs[39m> [32mINFO[39m RewardModule dependencies initialized 
2024-10-09T11:21:09.797Z <[95mnestjs[39m> [32mINFO[39m MonitorModule dependencies initialized 
2024-10-09T11:21:09.797Z <[95mnestjs[39m> [32mINFO[39m ProjectModule dependencies initialized 
2024-10-09T11:21:09.800Z <[95mnestjs[39m> [32mINFO[39m AdminController {/}: 
2024-10-09T11:21:09.801Z <[95mnestjs[39m> [32mINFO[39m Mapped {/env.js, GET} route 
2024-10-09T11:21:09.801Z <[95mnestjs[39m> [32mINFO[39m AgreementController {/agreements}: 
2024-10-09T11:21:09.802Z <[95mnestjs[39m> [32mINFO[39m Mapped {/agreements/:id, GET} route 
2024-10-09T11:21:09.802Z <[95mnestjs[39m> [32mINFO[39m MonitorController {/monitor}: 
2024-10-09T11:21:09.802Z <[95mnestjs[39m> [32mINFO[39m Mapped {/monitor, GET} route 
2024-10-09T11:21:09.802Z <[95mnestjs[39m> [32mINFO[39m Mapped {/monitor, POST} route 
2024-10-09T11:21:09.802Z <[95mnestjs[39m> [32mINFO[39m PrometheusController {/metrics}: 
2024-10-09T11:21:09.802Z <[95mnestjs[39m> [32mINFO[39m Mapped {/metrics, GET} route 
2024-10-09T11:21:09.802Z <[95mnestjs[39m> [32mINFO[39m StatsController {/stats}: 
2024-10-09T11:21:09.802Z <[95mnestjs[39m> [32mINFO[39m Mapped {/stats/:deploymentId/:timestamp, POST} route 
2024-10-09T11:21:09.802Z <[95mnestjs[39m> [32mINFO[39m Mapped {/stats, POST} route 
2024-10-09T11:21:09.802Z <[95mnestjs[39m> [32mINFO[39m Mapped {/stats/:deploymentId/:from/:to, GET} route 
query: SELECT "ProjectEntity"."id" AS "ProjectEntity_id", "ProjectEntity"."status" AS "ProjectEntity_status", "ProjectEntity"."chainType" AS "ProjectEntity_chainType", "ProjectEntity"."projectType" AS "ProjectEntity_projectType", "ProjectEntity"."hostType" AS "ProjectEntity_hostType", "ProjectEntity"."rateLimit" AS "ProjectEntity_rateLimit", "ProjectEntity"."nodeEndpoint" AS "ProjectEntity_nodeEndpoint", "ProjectEntity"."queryEndpoint" AS "ProjectEntity_queryEndpoint", "ProjectEntity"."serviceEndpoints" AS "ProjectEntity_serviceEndpoints", "ProjectEntity"."details" AS "ProjectEntity_details", "ProjectEntity"."manifest" AS "ProjectEntity_manifest", "ProjectEntity"."baseConfig" AS "ProjectEntity_baseConfig", "ProjectEntity"."advancedConfig" AS "ProjectEntity_advancedConfig", "ProjectEntity"."projectConfig" AS "ProjectEntity_projectConfig" FROM "project_entity" "ProjectEntity"
node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^

Error: connect ENOENT /var/run/docker.sock
    at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1278:16) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'connect',
  address: '/var/run/docker.sock'
}

@pamanseau pamanseau changed the title Unable to connect to the database in Kubernetes Kubernetes Compatibility Oct 11, 2024
@pamanseau
Copy link
Author

Branch #529

@pamanseau
Copy link
Author

Regarding the indexer-coordinator that uses Docker Compose TS libraries
As I see in the code and docs it looks like it's creating 2 new pods for each indexed subgraph and registering them on the network.

Kubernetes environment is different.

Each Subgraph needs:

  • Subql-indexer (substrate)
  • Subql-query

I would like more documentation or insights from you to see what in your opinion should be done to indexer-coordinator to work with Kubernetes.

@icezohu
Copy link
Collaborator

icezohu commented Nov 21, 2024 •
edited
Loading

just updated pr 529, try with this version: subquerynetwork/indexer-coordinator-dev:v2.8.1-100

@pamanseau
Copy link
Author

@icezohu Doing a follow up on this issue. Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pamanseau @icezohu