STIX Patterning needs a capability for setting and manipulating variables

[treyka]

No description provided.

[johnwunder]

Do you have an elaboration on this? Is it similar to this item? https://github.com/mitre/stix2patterns_translator/wiki/Property-Comparison-Proposal

[ikiril01]

@johnwunder I think it's the variables/backreferences stuff as we've described here: https://docs.google.com/document/d/15qD9KBQcVcY4FlG9n_VGhqacaeiLlNcQ7zVEjc8I3b4/edit#heading=h.vw8nwfvm4pvm

[treyka]

@johnwunder What @ikiril01 said...

[gtback]

From my comment in a pattern-matcher issue:

For the record, this is also why I'm extremely hesitant to add variables or any other features to the spec. Their meaning may be clear in isolation, but when combined with other features (like how the * and NOT combine in this case), it could lead to ambiguity and misinterpretation.

[treyka]

@gtback Given that we are going to add new features to the patterning spec, what would be your recommendation (from an implementer's perspective) as to how to avoid ambiguity?

One approach which I like is to put an optional pattern_version property on the STIX Indicator and use minor release numbers (a la 2.1.1) to allow vetting additions to the patterning spec in code. If you get a STIX 2.1 Indicator, you parse as a stock 2.1 pattern. If the pattern_version property is set and you're not expecting to interact with bleeding-edge patterning additions, you can just log an error and drop the input.

[gtback]

First, don't introduce changes that break old patterns. Every valid STIX 2.0 pattern should be a valid STIX 2.1 pattern. I don't expect any problems with this, but want to be explicit.

Second, I'm not talking about ambiguous STIX 2.x content (what versions/features are used in any particular pattern); I'm talking about ambiguity in how different features of the patterning specification interact. For example:

• Do arithmetic operations that work on timestamps affect how START/STOP or WITHIN clauses work? Maybe not, since START/STOP/WITHIN apply to the timestamp of the observation, while operations might only be allowed to apply to properties of the thing being observed. But this should be explicit.
• When casting data, does that affect how incompatible data types are handled, especially for comparison operators like <?
• Does is_null() or a NULL constant work with list elements ([*]), or the IN operator, etc.?

Adding new fields to STIX data types, or even new STIX data types (or adding a new Comparsion operator to patterns), makes a roughly linear addition to complexity (like adding a new element to a list). But adding new features to STIX patterns multiplies the complexity of code that evaluates patterns (like adding a new dimension; every new features must be considered in combination with every other feature, and subset of features).

Most of the pattern spec is devoted to how to express patterns, not how to evaluate them. There are admittedly a lot of statements about evaluating them as well, but they're scattered throughout the spec with no real guidance about how to combine them or how to resolve conflicting information between them (for example, the issue I linked to).

---

To more directly answer your question, I don't think there should be versions of the patterning spec released between versions of STIX (like 2.1.1). Being able to parse a STIX 2.1 indicator means you should be able to parse a STIX 2.1 pattern. We've already given up on any substantial interoperability guarantees (forward or backward) between STIX 2.0 and STIX 2.1; let's not exacerbate that within STIX 2.1.

[jmgnc]

@treyka I don't think this is a breaking change.

[jordan2175]

We talked about this on 2019-06-05 and we need more fleshed out use cases and propose that this be looked at again in 2.2 time frame.

[srrelitz2]

Reach out to OCA about STIX shifter to coordinate cross system interoperability