IMDS v1 fallback not being recognized (Windows)

[CodaBool]

Describe the bug
Only some CloudWatch metrics are being pushed by the agent

Steps to reproduce

1. use a windows 2016 server (build 14393)
2. set imds v2 optional
3. disable metadata tags
4. start a cloudwatch agent with multiple metrics.

What did you expect to see?
Metrics for memory and disk space.

What did you see instead?
I only see memory related metrics. I do not see any disk metrics

What version did you use?
Version: 1.300051.0b992

What config did you use?

{
    "agent": {
        "metrics_collection_interval": 60,
        "debug": true
    },
    "metrics": {
        "namespace": "CWAgent",
        "metrics_collected": {
            "LogicalDisk": {
                "measurement": [
                    "Free Megabytes",
                    "% Free Space"
                ],
                "resources": [
                    "*"
                ]
            },
            "Memory": {
                "measurement": [
                    "Available Bytes",
                    "% Committed Bytes In Use"
                ]
            }
        }
    }
}

Environment
OS: windows 2016 server (build 14393)

Additional context
I have another instance that is also a windows machine, same IAM role, same imds optional setting, same metadata tags disabled and yet it can push disk and memory

• this is in GovCloud

logs with debug=true

Which is the error message here:

amazon-cloudwatch-agent/translator/util/ec2util/ec2util.go

Line 115 in da53e1c

fmt.Println("D! could not get hostname without imds v1 fallback enable thus enable fallback")

there is also some error about an init path. Which I can't find results on Google for.

the machine has imds v2 set to optional

Comparing the similar working instance to the not working instance. I see they both make imds v1 requests since Cloudwatch does pick up NoToken requests.

very similar instance (also windows machine) which can provide disk metrics

instance which cannot provide disk metrics

I attempted a request to imds v1 using PS

Last debug step I can think of is downloading a packet analyzer and seeing what info I can gleam there.

[the-mann]

Hello @CodaBool. First off, thanks for the detailed post.

I am pretty confident that IMDS v1 is not the issue here. The concerning log line to me is the following:

D! metric init has error: Unable to parse the counter path. Check the format and syntax of the specified path.

CloudWatch Agent gets performance metrics from Windows Performance Monitor counters using Performance Data Helper (PDH), which is where that log line comes from.

Since you mentioned the issue is only happening on one host, I'm wondering if something is off with that host's environment... What would help us figure that out is if you could share two things:

1. the output of the following command:

typeperf -qx PhysicalDisk

This will show us all of the installed performance counters for the counter object PhysicalDisk (source)

2. Additionally, could you please share the contents of the following file? (source)

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent.toml

this will help us understand exactly what the agent is passing PDH when it's querying for a specific performance counter (i.e. LogicalDisk\% Free Space)

PS: I found this troubleshooting guide, which may be of help... (note: Telegraf is used in this codebase, and actually defines the windows performance counters).