Ambiguous glob semantics

[dunkelziffer]

Thanks for the fix in #1712

However, it seems to not fix globs in general. For example **.*a -> **.*b still generates new nodes.

I'm still not sure, whether I like the semantics of globs in D2. For me a glob matches existing things. A glob suddenly creating new things feels weird. Probably that's super useful within D2 and I just didn't run into any of these use cases, yet. But if globs create new nodes, how would you even avoid infinite recursion in all cases? This seems like a slippery slope to me.

I guess my issue here is that there is no clear delimiter where the glob ends and the code starts. I'll denote the "glob" with square brackets.

Let's say I actually want to create a node a for all existing containers. I'd want to write:

[*].a

Let's say I only wanted to match existing nodes a within containers. I'd want to write:

[*.a]

The bracket syntax does not exist in D2. These look the same in D2 currently and I would need to resort to filters or this "hacky" syntax

*.*a

This works as a temporary workaround, but it doesn't actually express my intent. I don't want to match nodes that "end in a". I want those that are "exactly a".

Maybe filters will provide a solution that covers all edge-cases, but please also consider that they are overly verbose.

Consider introducing an explicit syntax for "the glob ends here". Probably we don't even need brackets as a glob always needs to start at the beginning of the expression to my understanding.

So maybe you could even use something very unobtrusive like ..:

Create new nodes "a with content" as well as b and c connected by an edge

*..a.content
*..b -> *..c

# or (edges not possible in this version)
*..a {
  content
}

# or (curly braces terminate glob)
* {
  a.content
  b -> c
}

Match existing nodes a and add content, match existing nodes b and c and add edge:

*.a..content
*.b -> *.c # (end of expression terminates glob)

# or (curly braces terminate glob, edges not possible in this version)
*.a {
  content
}

# or (this would actually create a new glob `a` that would be nested inside the outer glob `*`.
# Think of nested for-loops. This would basically be an optional execution. If there is an `a`,
# then add content to it. Could be useful for templates where I don't know whether all code
# will be needed. D2 could internally combine globs that are separated by curly braces.)
* {
  a..content
  # looks weird
  b.. -> c..
  # maybe instead
  b? -> c?
}

I chose the .. syntax first, becaus it doesn't use any new symbols. If it doesn't clash with the rest of the syntax, we could also use ?.. That feels even more consistent with the edge syntax:

• Unconditionally create 2 nodes and an edge: b -> c
• If b exists, create c and a connecting edge: b? -> c
• If b and c exist, create an edge: b? -> c?

The ? would just mean "the glob ends here" and would be usable both for further chaining with . as well as for creating edges.

This would bring a new default behaviour:

• Text like a would not be a glob (unless explicitly turned into one by a.. or a?) and would always create a new node. I guess that's what a user expects
• Expressions that contain a * would by default be a glob as a whole (unless the user explicitly terminates the glob early with a ? or implicitly with curly braces) and would never create new nodes. I guess that's also reasonable
• I chose to make curly braces terminate the glob so code inside curly braces wouldn't need to know whether the outside expression was a glob or not. Probably splat imports should also terminate globs.

[alixander]

I understand the problem of targeting vs creating, and I understand that new syntax can disambiguate.

However, it's a design decision of the language to stick with readability. I think the .. and ? are reasonable proposals in a language that makes the decision in the other direction.

---

Let's say I want all inner node server to connect to load balancer.

**.server -> lb currently creates a server at each stop. So how should we specify that I only want to create edges, not nodes?

It's connections that have the property of creating when it doesn't exist, not globs. E.g., if you write a -> b, if shapes with those IDs don't exist, they will be created along with the connection. Globs should be consistent with that behavior. There just needs to be more specificity in what the glob captures.

I suppose any filter on the source would only match existing nodes.

**.server -> lb: {
  &src.label: *
}

That doesn't seem overly verbose.

[dunkelziffer]

My diagrams contain parts like these:

# define lots of containers with imports from my component library, e.g.
step_1 {
  forms/enter_credentials_1: ...@forms/enter_credentials
  # ... more similar imports
}

step_2 {
  processes/create_account_1: ...@processes/create_account
  # ... more similar imports
}

# ... more similar steps

# define lots of container-agnostic edges between imports:
*.*forms/enter_credentials_1.out.temporary_state -> *.*processes/create_account_1.in.temporary_state
# ... a lot more of these edges

I currently have 20+ occurrances of the last line. That's the old and small version of the diagram. The finished one will have close to 100 edges. I don't want to write the following 100 times:

*.forms/enter_credentials_1.out.temporary_state -> *.processes/create_account_1.in.temporary_state : {
  &src.label: *
  &dst.label: *
}

Do filters also work with imports? Then I could separate the block out into a separate file if_both_exist.d2 and only write:

*.forms/enter_credentials_1.out.temporary_state -> *.processes/create_account_1.in.temporary_state : @if_both_exist

That would be better, but still not optimal. Because if I would also want to style the edges, that would get complicated. Where do I put the styles in this version? An explicit syntax for the globs let's me use the import for the things that actually are important to me, e.g.

*?.forms/enter_credentials_1.out.temporary_state -> *?.processes/create_account_1.in.temporary_state : { class: [success] }
*?.forms/enter_credentials_1.out.temporary_state -> *?.forms/enter_credentials_1.in.temporary_state : { class: [error] }

EDIT:

Is the following possible?

*.forms/enter_credentials_1.out.temporary_state -> *.processes/create_account_1.in.temporary_state : { class: [success]; ...@if_both_exist }
*.forms/enter_credentials_1.out.temporary_state -> *.forms/enter_credentials_1.in.temporary_state : { class: [error]; ...@if_both_exist }

That would be getting close to acceptable. Still, it's a lot of redundancy, because I need this @if_both_exist for every single edge in my main file.

EDIT 2:

Can classes apply filters? Is the following possible

*.forms/enter_credentials_1.out.temporary_state -> *.processes/create_account_1.in.temporary_state : { class: [success; edge] }
*.forms/enter_credentials_1.out.temporary_state -> *.forms/enter_credentials_1.in.temporary_state : { class: [error; edge] }

With:

classes: {
  success.stroke: green
  error.stroke: red
  edge {
    &src.label: *
    &dst.label: *
  }
}

[dunkelziffer]

I tried the following in the playground:

a: {aa}
b: {bb}
*.aa -> *.bb: { class: [edge] }

classes: {
  edge: {
    &src.label: *
    &dst.label: *
  }
}

I got the error "glob filters cannot be used outside globs". This breaks both my attempted solutions, classes as well as imports. Could you make this error happen lazily? So if a block defines a glob filter and is only used within globs, could you make this not raise an error and work as intended? Also, it might be in better accordance with your desing goals (https://d2lang.com/tour/design#warnings--errors) to ignore these glob filters where not applicable and output a warning instead of raising an error and terminating.

EDIT:

I tried the following locally and even that doesn't work:

*.unique_name_1 -> *.unique_name_2: {
  &src.label: *
  &dst.label: *
  class: [success]
}

I also get "glob filters cannot be used outside globs". So your currently proposed solution is actually a syntax error right now.

One more question: are there any &src and &dst filters already implemented and working?