[Bug]: aws_elasticache must be in available state before modifying tags

[fredfrazao]

Terraform Core Version

1.5.1

AWS Provider Version

5.0

Affected Resource(s)

aws_elasticache_subnet_group
aws_elasticache_replication_group

Expected Behavior

Update tag successfully.

Actual Behavior

This scenario occurred due to a discrepancy in the tagging conventions between the old and new configurations, compounded by the behaviour of Terraform Enterprise (TFE) during resource updates.

Initially, when the resource was created the tag environment was set to "production", with the new configuration the tags that were applied were changed to Environment with the value PRODUCTION

When TFE executed, it recognized it needs to update existing resources based on the new tag configuration. Consequently, it attempted to update the aws_elasticache_subnet_group resource by modifying its tags. Specifically, it changed the environment tag to Environment and adjusted its value to "PRODUCTION", reflecting the updated tagging on the resource.

However, during the execution of the second step, which involved the aws_elasticache_replication_group resource, an error occurred. This error stemmed from the fact that the Elasticache cluster associated with this resource was not available at that moment to perform tagging update operations.

To fix this we had to re-run the TFE in order to update the tag

Relevant Error/Panic Output Snippet

InvalidReplicationGroupState: Cluster not in available state to perform tagging operations. status code: 400

Terraform Configuration Files

resource "aws_elasticache_subnet_group" "default" {
  name       = "${var.service_name}-cache-subnet"
  subnet_ids = var.vpc_private_subnet_ids
  tags       = var.service_tags
}

resource "aws_elasticache_replication_group" "default" {
  replication_group_id = "${var.service_name}-replication-group"
  description          = "Redis cluster for Gitlab application"
  security_group_ids   = [var.vpc_default_sg]
  subnet_group_name    = aws_elasticache_subnet_group.default.name

  snapshot_arns = var.snapshot_arns

  # single shard primary with read replica(s)
  num_cache_clusters         = var.num_cache_clusters
  automatic_failover_enabled = var.num_cache_clusters > 1 ? true : false
  multi_az_enabled           = var.num_cache_clusters > 1 ? true : false

  node_type            = var.instance_type
  port                 = var.redis_port
  parameter_group_name = var.parameter_group_name
  engine_version       = var.engine_version
  apply_immediately    = true

  at_rest_encryption_enabled = var.at_rest_encryption_enabled
  kms_key_id                 = var.cmk_arn

  transit_encryption_enabled = var.transit_encryption_enabled
  auth_token                 = data.aws_secretsmanager_secret_version.user.secret_string

  snapshot_retention_limit = var.redis_snapshot_retention_period
  snapshot_window          = var.redis_snapshot_window
  maintenance_window       = var.redis_maintenance_window
  tags                     = var.service_tags

}

Steps to Reproduce

Set a tags environment = production apply , then change the tag configuration to Environment = PRODUCTION and re apply

Debug Output

"@Level":"error","@message":"Error: updating tags for ElastiCache Replication Group (arn:XXXX): tagging resource (arnXXX): InvalidReplicationGroupState: Cluster not in available state to perform tagging operations.\n\tstatus code: 400, request id: XXXXX","@module":"terraform.ui","@timestamp":"2023-11-29T18:15:03.252125Z","diagnostic":{"severity":"error","summary":"updating tags for ElastiCache Replication Group (arn:XXXXX): tagging resource (arn:XXX): InvalidReplicationGroupState: Cluster not in available state to perform tagging operations.\n\tstatus code: 400, request id: XXXX","detail":"","address":"module.redis-cache.aws_elasticache_replication_group.default","range":{"filename":".terraform/modules/redis-cache/modules/redis-cache/main.tf","start":{"line":41,"column":56,"byte":1614},"end":

Panic Output

No response

Important Factoids

In this case the depends_on meta-argument to handle this situations would not help, because it waits for the resource to be created but not ready

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[cblkwell]

I'm also running into this issue; nothing is changing on the resource for us but the tagging, and it's failing with this error. It looks like it might be an intermittent problem though; I can re-run the plan/apply and get it work sometimes.

I'm not using TFE (we use Atlantis instead), so I am pretty sure this is just a provider issue.

[unfor19]

@cblkwell Thanks for pointing out that it's intermittent (random) issue. I also retried and suddenly it worked with no issues

[Snojo]

Confirmed intermittent issue. Rerunning the terraform managed to apply on second try.