Akka runtime: KeepRunningWithWorker sent to dead letter

[hamsterready]

Hi,

Thanks for this project. It looks very promising.

I tried to use Akka runtime and in single node setup, using config like this:

zio.entity {
  akka-runtime {
    number-of-shards = 5
    ask-timeout = 10s
    idle-timeout = 10s
  }
}
akka {
  actor {
    provider = cluster
  }
  remote.artery {
    canonical {
        hostname = "127.0.0.1"
        port = 2551
        port = ${?AKKA_PORT}
    }
  }
  cluster {
    seed-nodes = [ "akka://[email protected]:2551" ]
    downing-provider-class = "akka.cluster.sbr.SplitBrainResolverProvider"
  }
  actor {
    serialization-identifiers {
      "zio.entity.runtime.akka.serialization.MessageSerializer" = 140
    }
    serializers {
      "zio.entity.akka-message-serializer" = "zio.entity.runtime.akka.serialization.MessageSerializer"
    }
    serialization-bindings {
      "zio.entity.runtime.akka.serialization.Message" = "zio.entity.akka-message-serializer"
    }
  }
}

akka.actor.allow-java-serialization = "on"

akka {
  loglevel = "INFO"
}

I am getting:

[INFO] [05/09/2022 12:47:38.249] [zio-default-async-23] [ArteryTransport(akka://Hello)] Remoting started with transport [Artery tcp]; listening on address [akka://[email protected]:2551] with UID [6396245429081092268]
[INFO] [05/09/2022 12:47:38.267] [zio-default-async-23] [Cluster(akka://Hello)] Cluster Node [akka://[email protected]:2551] - Starting up, Akka version [2.6.17] ...
[INFO] [05/09/2022 12:47:38.357] [zio-default-async-23] [Cluster(akka://Hello)] Cluster Node [akka://[email protected]:2551] - Registered cluster JMX MBean [akka:type=Cluster]
[INFO] [05/09/2022 12:47:38.357] [zio-default-async-23] [Cluster(akka://Hello)] Cluster Node [akka://[email protected]:2551] - Started up successfully
[INFO] [05/09/2022 12:47:38.406] [Hello-akka.actor.default-dispatcher-15] [akka://Hello/system/cluster/core/daemon/downingProvider] SBR started. Config: strategy [KeepMajority], stable-after [20 seconds], down-all-when-unstable [15 seconds], selfUniqueAddress [akka://[email protected]:2551#6396245429081092268], selfDc [default].
[INFO] [akkaMemberChanged][05/09/2022 12:47:38.407] [Hello-akka.actor.internal-dispatcher-13] [Cluster(akka://Hello)] Cluster Node [akka://[email protected]:2551] - Node [akka://[email protected]:2551] is JOINING itself (with roles [dc-default], version [0.0.0]) and forming new cluster
[INFO] [05/09/2022 12:47:38.409] [Hello-akka.actor.internal-dispatcher-13] [Cluster(akka://Hello)] Cluster Node [akka://[email protected]:2551] - is the new leader among reachable nodes (more leaders may exist)
[INFO] [akkaMemberChanged][05/09/2022 12:47:38.414] [Hello-akka.actor.internal-dispatcher-13] [Cluster(akka://Hello)] Cluster Node [akka://[email protected]:2551] - Leader is moving node [akka://[email protected]:2551] to [Up]
[INFO] [05/09/2022 12:47:38.427] [Hello-akka.actor.default-dispatcher-15] [akka://Hello/system/cluster/core/daemon/downingProvider] This node is now the leader responsible for taking SBR decisions among the reachable nodes (more leaders may exist).
[INFO] [05/09/2022 12:47:38.549] [Hello-akka.actor.internal-dispatcher-6] [akka://[email protected]:2551/system/sharding/Campaign] Campaign: Idle entities will be passivated after [2.000 min]
[INFO] [akkaClusterSingletonStarted][05/09/2022 12:47:38.561] [Hello-akka.actor.internal-dispatcher-2] [akka://[email protected]:2551/system/sharding/CampaignCoordinator] Singleton manager starting singleton actor [akka://Hello/system/sharding/CampaignCoordinator/singleton]
[INFO] [05/09/2022 12:47:38.561] [Hello-akka.actor.internal-dispatcher-2] [akka://[email protected]:2551/system/sharding/CampaignCoordinator] ClusterSingletonManager state change [Start -> Oldest]
[INFO] [05/09/2022 12:47:38.573] [Hello-akka.actor.default-dispatcher-4] [akka://[email protected]:2551/system/sharding/CampaignCoordinator/singleton/coordinator] Campaign: ShardCoordinator was moved to the active state with [0] shards
2022-05-09_12:47:38.642 [zio-default-async-13] INFO  r.c.g.GRPCServer$ - Staring GRPC server at port 9000 
2022-05-09_12:47:38.652 [zio-default-async-13] INFO  r.c.h.HttpServer$ - Staring HTTP server at 8000 
Server started on port: 8000
[INFO] [05/09/2022 12:47:38.895] [Hello-akka.actor.internal-dispatcher-6] [akka://[email protected]:2551/system/sharding/RunningCampaignsTracker] RunningCampaignsTracker: Idle entities will be passivated after [2.000 min]
[INFO] [akkaClusterSingletonStarted][05/09/2022 12:47:38.897] [Hello-akka.actor.internal-dispatcher-14] [akka://[email protected]:2551/system/sharding/RunningCampaignsTrackerCoordinator] Singleton manager starting singleton actor [akka://Hello/system/sharding/RunningCampaignsTrackerCoordinator/singleton]
[INFO] [05/09/2022 12:47:38.897] [Hello-akka.actor.internal-dispatcher-14] [akka://[email protected]:2551/system/sharding/RunningCampaignsTrackerCoordinator] ClusterSingletonManager state change [Start -> Oldest]
[INFO] [05/09/2022 12:47:38.899] [Hello-akka.actor.default-dispatcher-4] [akka://[email protected]:2551/system/sharding/RunningCampaignsTrackerCoordinator/singleton/coordinator] RunningCampaignsTracker: ShardCoordinator was moved to the active state with [0] shards
[INFO] [05/09/2022 12:47:39.207] [Hello-akka.actor.default-dispatcher-15] [akka://[email protected]:2551/system/sharding/RunningCampaignsTracker/0/0/worker] [0] Starting process RunningCampaignsTracker
[INFO] [05/09/2022 12:47:39.274] [Hello-akka.actor.default-dispatcher-4] [akka://[email protected]:2551/system/sharding/RunningCampaignsTracker/0/0/worker] [0] Process started RunningCampaignsTracker
[INFO] [akkaDeadLetter][05/09/2022 12:47:40.916] [Hello-akka.actor.default-dispatcher-15] [akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker] Message [zio.entity.runtime.akka.readside.ReadSideWorkerActor$KeepRunningWithWorker] from Actor[akka://Hello/user/ReadSideSupervisor-RunningCampaignsTracker#2097191090] to Actor[akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker#2128380693] was unhandled. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [akkaDeadLetter][05/09/2022 12:47:42.933] [Hello-akka.actor.default-dispatcher-15] [akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker] Message [zio.entity.runtime.akka.readside.ReadSideWorkerActor$KeepRunningWithWorker] from Actor[akka://Hello/user/ReadSideSupervisor-RunningCampaignsTracker#2097191090] to Actor[akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker#2128380693] was unhandled. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [akkaDeadLetter][05/09/2022 12:47:44.952] [Hello-akka.actor.default-dispatcher-15] [akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker] Message [zio.entity.runtime.akka.readside.ReadSideWorkerActor$KeepRunningWithWorker] from Actor[akka://Hello/user/ReadSideSupervisor-RunningCampaignsTracker#2097191090] to Actor[akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker#2128380693] was unhandled. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [akkaDeadLetter][05/09/2022 12:47:46.973] [Hello-akka.actor.default-dispatcher-15] [akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker] Message [zio.entity.runtime.akka.readside.ReadSideWorkerActor$KeepRunningWithWorker] from Actor[akka://Hello/user/ReadSideSupervisor-RunningCampaignsTracker#2097191090] to Actor[akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker#2128380693] was unhandled. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [akkaDeadLetter][05/09/2022 12:47:48.992] [Hello-akka.actor.default-dispatcher-4] [akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker] Message [zio.entity.runtime.akka.readside.ReadSideWorkerActor$KeepRunningWithWorker] from Actor[akka://Hello/user/ReadSideSupervisor-RunningCampaignsTracker#2097191090] to Actor[akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker#2128380693] was unhandled. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [akkaDeadLetter][05/09/2022 12:47:51.012] [Hello-akka.actor.default-dispatcher-4] [akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker] Message [zio.entity.runtime.akka.readside.ReadSideWorkerActor$KeepRunningWithWorker] from Actor[akka://Hello/user/ReadSideSupervisor-RunningCampaignsTracker#2097191090] to Actor[akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker#2128380693] was unhandled. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [akkaDeadLetter][05/09/2022 12:47:53.033] [Hello-akka.actor.default-dispatcher-4] [akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker] Message [zio.entity.runtime.akka.readside.ReadSideWorkerActor$KeepRunningWithWorker] from Actor[akka://Hello/user/ReadSideSupervisor-RunningCampaignsTracker#2097191090] to Actor[akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker#2128380693] was unhandled. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [akkaDeadLetter][05/09/2022 12:47:55.052] [Hello-akka.actor.default-dispatcher-15] [akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker] Message [zio.entity.runtime.akka.readside.ReadSideWorkerActor$KeepRunningWithWorker] from Actor[akka://Hello/user/ReadSideSupervisor-RunningCampaignsTracker#2097191090] to Actor[akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker#2128380693] was unhandled. [8] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [akkaDeadLetter][05/09/2022 12:47:57.071] [Hello-akka.actor.default-dispatcher-15] [akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker] Message [zio.entity.runtime.akka.readside.ReadSideWorkerActor$KeepRunningWithWorker] from Actor[akka://Hello/user/ReadSideSupervisor-RunningCampaignsTracker#2097191090] to Actor[akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker#2128380693] was unhandled. [9] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [akkaDeadLetter][05/09/2022 12:47:59.093] [Hello-akka.actor.default-dispatcher-16] [akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker] Message [zio.entity.runtime.akka.readside.ReadSideWorkerActor$KeepRunningWithWorker] from Actor[akka://Hello/user/ReadSideSupervisor-RunningCampaignsTracker#2097191090] to Actor[akka://Hello/system/sharding/RunningCampaignsTracker/0/0/worker#2128380693] was unhandled. [10] dead letters encountered, no more dead letters will be logged in next [5.000 min]. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

In a multi node setup it looks like when I start my read-side on each node only one is active at the same time (I am using different consumer id on each node of the cluster).