Handling Comparison Expressions that Cannot Be Evaluated

[ikiril01]

We need to determine how we treat Comparison Expressions in STIX Patterning that cannot be evaluated for whatever reason (e.g, if a product cannot evaluate a pattern because it refers to data that the product does not handle).

Here's the original text that Jason proposed:

​Comparison Expressions may include one or more Object Path(s) not obtainable within a given Observation, either because the Object Path(s) are not present within the Observation or because the evaluating entity does not have access to the data necessary in order to evaluate the specified Object Path(s). When evaluating a Comparison Expression against one or more Object Path(s) that are not present or whose data cannot be obtained, the Comparison Expression MUST evaluate to true, regardless of the operators included in the expression.

And here's some clarifying explanation from Jason:

"Say I get a pattern that says "[ file that has this hash ] and [ this registry key ]", and I can inspect files live on the wire but I do not have an endpoint system to read registry keys. Should I throw an error and do nothing, or return all the matches for A? I would argue most users want the latter behavior.. which means B should evaluate to True"

However, there are a number of other questions here, including:

1. Should we be oriented towards having more false positives (MUST evaluate to true) or false negatives (MUST evaluate to false)?
2. Should we have another condition (e.g., error) if a pattern cannot be evaluated?
3. Should we allow for a "flag" on the pattern that dictates how it should be handled in this case? E.g., return_true_unsupported_pattern: { true | false }

[allant0]

In a similar security use case, firewalls that have errors have a config option to fall open or fall closed so that if the software crashes on the firewall there is a predictable behavior in the firewall. In some customer environments they choose fall open because of the business priority that they are making is allow traffic over security concerns whereas others choose fall closed and therefore if security is compromised then shut down everything. Both are legitimate paths and are based on business policy not technical specification.

I share this analogy because I think it has relevance to this particular issue.

If a pattern is not supported by a product that receives the pattern then its really up to the product+policy on how that pattern is treated in that situation. It cannot be mandated by a provider of the pattern.

Therefore if we add a property to suggest how patterns are treated then it should be a SHOULD only and not a must.

I personally think a more useful property to add would be a descriptor that helps products determine without parsing the pattern what components the pattern relates to and whether the pattern would apply to the product in the first place.

For example:

pattern_components : { endpoint || network-packet || memory || network-state || applications }
os-list: { ... list of OSes the pattern should apply to }

Then it will be easier for recipient to determine whether a pattern applies to it or not immediately and could avoid parsing and then failing to work.

[gtback]

We need to be careful about mixing specification about how patterns are expressed with how they are evaluated. The latter definitely depends on the former, but there's a lot more variability between products/vendors/use-cases that we should be hesitant to overspecify.

There's definitely room for some guidance in the interop spec, or some other guide, but saying "to support STIX patterns (and therefore STIX), you MUST do X with patterns" seems like a bad idea.

[JasonKeirstead]

@gtback is right. The root problem here and the reason I cooked up this text in the first place is because at the end of the day, any syntactically valid pattern (which these are) has to evaluate to either True or False, so we need to figure out how that case is handled.

Individual product guidance in Interop can be given stating that the object paths inside patterns can (and normally should) be evaluated before executing the pattern, allowing products to decide how to handle this scenario themselves (they can elect to error out or skip the evaluation easily at that point).

But if a product does decide to continue with the pattern, then we need to specify how it should be evaluated. "Its an error condition" is not an option, because unlike TAXII, this is not a protocol spec and we have no way to communicate errors. The pattern is syntactically valid.

[gtback]

@JasonKeirstead I'm assuming you mean:

any syntactically valid pattern (which these are) has to evaluate to either True or False on a given set of input data.

(patterns aren't True or False on their own 😉)

Still, I'm not sure I agree with your assertion.

1. In the first place, I don't know how we validate/verify matching behavior. The pattern-matcher operates on STIX observed-data instances, but only because that is the only common denominator; I assume most programs that interpret patterns will be matching against data in some native form (hence the need for this issue!).
2. At most, we can say something like "given a file named foo.exe, the pattern [file:name = 'foo.exe'] MUST NOT evaluate to False (and similar for situations which MUST NOT evaluate to True). But I see no reason why we can't allow implementations to return "Unknown" or various error conditions.
3. The interop spec could potentially be more rigid, but as @JasonKeirstead said, STIX is not a protocol spec, so what a consumer does when they receive a pattern is mostly outside the scope of STIX.

[JasonKeirstead]

@gtback Yes, on a given set of data, a pattern has to return either True or False. "Error" isn't really a valid thing to define in the STIX spec because its not a protocol. A pattern is either valid STIX or it is not valid STIX there isn't really a middle ground.

For #1 - Lets leave "how to do validation" out - that is outside the scope of a data spec. That's an interop discussion.

For #2 - We can't say you can "return Unknown", because this is not a protocol spec. Patterns don't "return" anything.

[gtback]

By "return Unknown", I just meant that a tool that evaluates patterns is "allowed" to say "I can't evaluate this pattern", based on any number of factors (unsupported observable types being just one example).

[JasonKeirstead]

@gtback A tool is allowed to do anything it wants.. the spec doesn't dictate how a tool behaves.

[gtback]

the spec doesn't dictate how a tool behaves.

Isn't that what we're talking about in this issue? Dictating how tools should behave?

[JasonKeirstead]

@gtback No..... the STIX spec does not dictate how tools behave. It dictates a data format. What someone does with that data is totally up to them. I could build a tool that is STIX 2 Preferred that takes IOCs and 3D prints jenga blocks with STIX patterns on them. Totally valid and STIX compliant.

[gtback]

So should this issue not be closed as out of scope for the STIX spec?

[allant0]

The STIX spec has compliance statements. If any compliance statement is made about pattern interpretation or support within a product then those compliance statements need to be correct otherwise implementations will not be compliant.

Are there any normative statements on how a product would return false or true for unsupported patterns? If not, then the behavior is undefined. But if there are any normative statements on how pattern results should be interpreted then its defined and we need to make sure its 'correct'.

[JasonKeirstead]

In my opinion, if we don't define this at all, then consumers of STIX products in the market aren't going to have any idea how their products should consume indicators that test on custom properties. If we don't put something in the normative text about the expected behaviour, we can't tackle it in Interop either.

I think we need to separate "how a tool should behave" from "how should a pattern evaluate". These are to me two completely different questions. The purpose of a data interchange specification is not to dictate how tools should behave, it is to dictate how a piece of content should be interpreted when consumed by anything (be that tool, or even a person).

@allant0 As to your question, no there are no compliance statements in the spec about returning true or false, because patterns aren' defined to return anything at all. We can't write a compliance statement like that without making assumptions as to how STIX is being used in code.

Any half-decently written tool IMO is going to pre-test an expression and do something if it can't parse or handle something in it properly (log a warning, error out, skip it, etc). That doesn't mean that we shouldn't bother putting this into the spec.

[JasonKeirstead]

I stand by my original suggestion in the Github issue as the suggestion

[jordan2175]

We talked about this on 2019-06-05 and Allan suggested that we change the text to evaluate as false.

[jordan2175]

Text was added in 9.6