From ac9f349e8e76a9478483be9c7fe4e22a4d60d02e Mon Sep 17 00:00:00 2001 From: Phil Rzewski Date: Fri, 24 Jan 2025 13:32:09 -0800 Subject: [PATCH] Convert more operator examples to mdtest-spq --- docs/language/operators/get.md | 2 +- docs/language/operators/head.md | 37 +++++---- docs/language/operators/join.md | 12 ++- docs/language/operators/load.md | 12 +-- docs/language/operators/merge.md | 16 ++-- docs/language/operators/over.md | 117 ++++++++++++++++------------ docs/language/operators/pass.md | 34 ++++---- docs/language/operators/put.md | 60 +++++++++------ docs/language/operators/rename.md | 64 ++++++++------- docs/language/operators/sample.md | 35 ++++++--- docs/language/operators/search.md | 124 +++++++++++++++++++----------- 11 files changed, 310 insertions(+), 203 deletions(-) diff --git a/docs/language/operators/get.md b/docs/language/operators/get.md index ca9bc6987b..e36f2fe4b3 100644 --- a/docs/language/operators/get.md +++ b/docs/language/operators/get.md @@ -4,4 +4,4 @@ ### Synopsis -`get` is a shorthand notation for `from`. See the [from operator](from.md) documentation for details. +`get` is a shorthand notation for `from`. See the [`from` operator](from.md) documentation for details. diff --git a/docs/language/operators/head.md b/docs/language/operators/head.md index 319c2fc03f..556ef7f458 100644 --- a/docs/language/operators/head.md +++ b/docs/language/operators/head.md @@ -17,30 +17,37 @@ is not provided, the value of N defaults to `1`. ### Examples _Grab first two values of arbitrary sequence_ -```mdtest-command -echo '1 "foo" [1,2,3]' | super -z -c 'head 2' - -``` - -```mdtest-output +```mdtest-spq +# spq +head 2 +# input +1 +"foo" +[1,2,3] +# expected output 1 "foo" ``` _Grab first two values of arbitrary sequence, using a different representation of two_ -```mdtest-command -echo '1 "foo" [1,2,3]' | super -z -c 'head 1+1' - -``` - -```mdtest-output +```mdtest-spq +# spq +head 1+1 +# input +1 +"foo" +[1,2,3] +# expected output 1 "foo" ``` _Grab the first record of a record sequence_ -```mdtest-command -echo '{a:"hello"}{b:"world"}' | super -z -c head - -``` - -```mdtest-output +```mdtest-spq +# spq +head +# input +{a:"hello"} +# expected output {a:"hello"} ``` diff --git a/docs/language/operators/join.md b/docs/language/operators/join.md index e92e7ccc00..202ff58274 100644 --- a/docs/language/operators/join.md +++ b/docs/language/operators/join.md @@ -18,7 +18,7 @@ The first `join` syntax shown above was more recently introduced and is in some ways similar to other languages such as SQL. The second was the original `join` -syntax in SuperPipe. Most joins can be expressed using either syntax. See the +syntax in the language. Most joins can be expressed using either syntax. See the [join tutorial](../../tutorials/join.md) for details. @@ -40,9 +40,13 @@ The available join types are: For anti join, the `` is undefined and thus cannot be specified. -> Currently, only exact equi-join is supported and join keys must be field -> expressions. A future version of join will have more flexible join -> expressions. +{{% tip "Note" %}} + +Currently, only exact equi-join is supported and join keys must be field +expressions. A future version of `join` will have more flexible join +expressions. + +{{% /tip %}} ### Examples diff --git a/docs/language/operators/load.md b/docs/language/operators/load.md index e726962c39..6a5c6daee1 100644 --- a/docs/language/operators/load.md +++ b/docs/language/operators/load.md @@ -44,8 +44,8 @@ echo '{flip:1,result:"heads"} {flip:2,result:"tails"}' | super db -q create -orderby flip:asc bigflips super db query -f text ' from :branches - |> yield pool.name + "@" + branch.name - |> sort' + | yield pool.name + "@" + branch.name + | sort' ``` The lake then contains the two pools: @@ -62,8 +62,8 @@ _Modify some values, load them into the `main` branch of our empty `bigflips` po ```mdtest-command super db -lake example query ' from coinflips - |> result:=upper(result) - |> load bigflips + | result:=upper(result) + | load bigflips ' > /dev/null super db -lake example query -z 'from bigflips' @@ -78,8 +78,8 @@ _Add a filtered subset of records to our `onlytails` branch, while also adding m ```mdtest-command super db -lake example query ' from coinflips - |> result=="tails" - |> load coinflips@onlytails + | result=="tails" + | load coinflips@onlytails author "Steve" message "A subset" meta "\"Additional metadata\"" diff --git a/docs/language/operators/merge.md b/docs/language/operators/merge.md index da0e6944ea..4dc23024ae 100644 --- a/docs/language/operators/merge.md +++ b/docs/language/operators/merge.md @@ -17,11 +17,17 @@ where the values from the upstream pipeline branches are forwarded based on thes ### Examples _Copy input to two pipeline branches and merge_ -```mdtest-command -echo '1 2' | super -z -c 'fork (=>pass =>pass) |> merge this' - -``` - -```mdtest-output +```mdtest-spq +# spq +fork ( + =>pass + =>pass +) +| merge this +# input +1 +2 +# expected output 1 1 2 diff --git a/docs/language/operators/over.md b/docs/language/operators/over.md index f16781d8cb..83e6acd14e 100644 --- a/docs/language/operators/over.md +++ b/docs/language/operators/over.md @@ -31,94 +31,113 @@ The nested subquery depicted as `` is called a [lateral subquery](../la ### Examples _Over evaluates each expression and emits it_ -```mdtest-command -echo null | super -z -c 'over 1,2,"foo"' - -``` - -```mdtest-output +```mdtest-spq +# spq +over 1,2,"foo" +# input +null +# expected output 1 2 "foo" ``` -_The over clause is evaluated once per each input value_ -```mdtest-command -echo "null null" | super -z -c 'over 1,2' - -``` -```mdtest-output +_The over clause is evaluated once per each input value_ +```mdtest-spq +# spq +over 1,2 +# input +null +null +# expected output 1 2 1 2 ``` -_Array elements are enumerated_ -```mdtest-command -echo null | super -z -c 'over [1,2],[3,4,5]' - -``` -```mdtest-output +_Array elements are enumerated_ +```mdtest-spq +# spq +over [1,2],[3,4,5] +# input +null +# expected output 1 2 3 4 5 ``` -_Over traversing an array_ -```mdtest-command -echo '{a:[1,2,3]}' | super -z -c 'over a' - -``` -```mdtest-output +_Over traversing an array_ +```mdtest-spq +# spq +over a +# input +{a:[1,2,3]} +# expected output 1 2 3 ``` -_Filter the traversed values_ - -```mdtest-command -echo '{a:[6,5,4]} {a:[3,2,1]}' | super -z -c 'over a |> this % 2 == 0' - -``` -```mdtest-output +_Filter the traversed values_ +```mdtest-spq +# spq +over a | this % 2 == 0 +# input +{a:[6,5,4]} +{a:[3,2,1]} +# expected output 6 4 2 ``` -_Aggregate the traversed values_ - -```mdtest-command -echo '{a:[1,2]} {a:[3,4,5]}' | super -z -c 'over a |> sum(this)' - -``` -```mdtest-output +_Aggregate the traversed values_ +```mdtest-spq +# spq +over a | sum(this) +# input +{a:[1,2]} +{a:[3,4,5]} +# expected output 15 ``` -_Aggregate the traversed values in a lateral query_ -```mdtest-command -echo '{a:[1,2]} {a:[3,4,5]}' | super -z -c 'over a => ( sum(this) )' - -``` -```mdtest-output +_Aggregate the traversed values in a lateral query_ +```mdtest-spq +# spq +over a => ( sum(this) ) +# input +{a:[1,2]} +{a:[3,4,5]} +# expected output 3 12 ``` -_Access the outer values in a lateral query_ -```mdtest-command -echo '{a:[1,2],s:"foo"} {a:[3,4,5],s:"bar"}' | - super -z -c 'over a with s => (sum(this) |> yield {s,sum:this})' - -``` -```mdtest-output +_Access the outer values in a lateral query_ +```mdtest-spq +# spq +over a with s => (sum(this) | yield {s,sum:this}) +# input +{a:[1,2],s:"foo"} +{a:[3,4,5],s:"bar"} +# expected output {s:"foo",sum:3} {s:"bar",sum:12} ``` -_Traverse a record by flattening it_ -```mdtest-command -echo '{s:"foo",r:{a:1,b:2}} {s:"bar",r:{a:3,b:4}} ' | - super -z -c 'over flatten(r) with s => (yield {s,key:key[0],value})' - -``` -```mdtest-output +_Traverse a record by flattening it_ +```mdtest-spq +# spq +over flatten(r) with s => (yield {s,key:key[0],value}) +# input +{s:"foo",r:{a:1,b:2}} +{s:"bar",r:{a:3,b:4}} +# expected output {s:"foo",key:"a",value:1} {s:"foo",key:"b",value:2} {s:"bar",key:"a",value:3} diff --git a/docs/language/operators/pass.md b/docs/language/operators/pass.md index 38f0ef582a..c353926d6f 100644 --- a/docs/language/operators/pass.md +++ b/docs/language/operators/pass.md @@ -16,27 +16,31 @@ with operators that handle multiple branches of the pipeline such as ### Examples _Copy input to output_ -```mdtest-command -echo '1 2 3' | super -z -c pass - -``` - -```mdtest-output +```mdtest-spq +# spq +pass +# input +1 +2 +3 +# expected output 1 2 3 ``` _Copy each input value to three parallel pipeline branches and leave the values unmodified on one of them_ -```mdtest-command -echo '"HeLlo, WoRlD!"' | super -z -c ' - fork ( - => pass - => upper(this) - => lower(this) -) |> sort' - -``` - -```mdtest-output +```mdtest-spq +# spq +fork ( + => pass + => upper(this) + => lower(this) +) +| sort +# input +"HeLlo, WoRlD!" +# expected output "HELLO, WORLD!" "HeLlo, WoRlD!" "hello, world!" diff --git a/docs/language/operators/put.md b/docs/language/operators/put.md index 2426e3387c..b707feb909 100644 --- a/docs/language/operators/put.md +++ b/docs/language/operators/put.md @@ -44,43 +44,53 @@ yield {...this, : [, :...]} ### Examples _A simple put_ -```mdtest-command -echo '{a:1,b:2}' | super -z -c 'put c:=3' - -``` - -```mdtest-output +```mdtest-spq +# spq +put c:=3 +# input +{a:1,b:2} +# expected output {a:1,b:2,c:3} ``` -_The `put` keyword may be omitted_ -```mdtest-command -echo '{a:1,b:2}' | super -z -c 'c:=3' - -``` -```mdtest-output +_The `put` keyword may be omitted_ +```mdtest-spq +# spq +c:=3 +# input +{a:1,b:2} +# expected output {a:1,b:2,c:3} ``` -_A `put` operation can also be done with a record literal_ -```mdtest-command -echo '{a:1,b:2}' | super -z -c 'yield {...this, c:3}' - -``` -```mdtest-output +_A `put` operation can also be done with a record literal_ +```mdtest-spq +# spq +yield {...this, c:3} +# input +{a:1,b:2} +# expected output {a:1,b:2,c:3} ``` -_Missing fields show up as missing errors_ -```mdtest-command -echo '{a:1,b:2,c:3}' | super -z -c 'put d:=e' - -``` -```mdtest-output +_Missing fields show up as missing errors_ +```mdtest-spq +# spq +put d:=e +# input +{a:1,b:2,c:3} +# expected output {a:1,b:2,c:3,d:error("missing")} ``` -_Non-record input values generate errors_ -```mdtest-command -echo '{a:1} 1' | super -z -c 'b:=2' - -``` -```mdtest-output +_Non-record input values generate errors_ +```mdtest-spq {data-layout="stacked"} +# spq +b:=2 +# input +{a:1} +1 +# expected output {a:1,b:2} error({message:"put: not a record",on:1}) ``` diff --git a/docs/language/operators/rename.md b/docs/language/operators/rename.md index f595b5dc3b..bb276c15ab 100644 --- a/docs/language/operators/rename.md +++ b/docs/language/operators/rename.md @@ -28,53 +28,65 @@ and the error is emitted. ### Examples _A simple rename_ -```mdtest-command -echo '{a:1,b:2}' | super -z -c 'rename c:=b' - -``` - -```mdtest-output +```mdtest-spq +# spq +rename c:=b +# input +{a:1,b:2} +# expected output {a:1,c:2} ``` -_Nested rename_ -```mdtest-command -echo '{a:1,r:{b:2,c:3}}' | super -z -c 'rename r.a:=r.b' - -``` -```mdtest-output +_Nested rename_ +```mdtest-spq +# spq +rename r.a:=r.b +# input +{a:1,r:{b:2,c:3}} +# expected output {a:1,r:{a:2,c:3}} ``` + _Trying to mutate records with rename produces a compile-time error_ ```mdtest-command fails echo '{a:1,r:{b:2,c:3}}' | super -z -c 'rename w:=r.b' - ``` - +=> ```mdtest-output left-hand side and right-hand side must have the same depth (w vs r.b) at line 1, column 8: rename w:=r.b ~~~~~~ ``` -_Record literals can be used instead of rename for mutation_ -```mdtest-command -echo '{a:1,r:{b:2,c:3}}' | super -z -c 'yield {a,r:{c:r.c},w:r.b}' - -``` -```mdtest-output +_Record literals can be used instead of rename for mutation_ +```mdtest-spq +# spq +yield {a,r:{c:r.c},w:r.b} +# input +{a:1,r:{b:2,c:3}} +# expected output {a:1,r:{c:3},w:2} ``` -_Alternatively, mutations can be more generic and use drop_ -```mdtest-command -echo '{a:1,r:{b:2,c:3}}' | super -z -c 'yield {a,r,w:r.b} |> drop r.b' - -``` -```mdtest-output +_Alternatively, mutations can be more generic and use drop_ +```mdtest-spq +# spq +yield {a,r,w:r.b} |> drop r.b +# input +{a:1,r:{b:2,c:3}} +# expected output {a:1,r:{c:3},w:2} ``` -_Duplicate fields create structured errors_ -```mdtest-command -echo '{b:1} {a:1,b:1} {c:1}' | super -z -c 'rename a:=b' - -``` -```mdtest-output +_Duplicate fields create structured errors_ +```mdtest-spq {data-layout="stacked"} +# spq +rename a:=b +# input +{b:1} +{a:1,b:1} +{c:1} +# expected output {a:1} error({message:"rename: duplicate field: \"a\"",on:{a:1,b:1}}) {c:1} diff --git a/docs/language/operators/sample.md b/docs/language/operators/sample.md index 0dad1aee94..8dcaf6fa50 100644 --- a/docs/language/operators/sample.md +++ b/docs/language/operators/sample.md @@ -10,7 +10,7 @@ sample [] The `sample` operator is a syntactic shortcut for ``` -val:=any() by typeof() |> yield val +val:=any() by typeof() | yield val ``` If `` is not provided, `this` is used. @@ -22,23 +22,34 @@ through it all to slice and dice it. ### Examples _A simple sample_ -```mdtest-command -echo '1 2 3 "foo" "bar" 10.0.0.1 10.0.0.2' | super -z -c 'sample |> sort this' - -``` - -```mdtest-output +```mdtest-spq +# spq +sample | sort this +# input +1 +2 +3 +"foo" +"bar" +10.0.0.1 +10.0.0.2 +# expected output 1 "foo" 10.0.0.1 ``` _Sampling record shapes_ -```mdtest-command -echo '{a:1}{a:2}{s:"foo"}{s:"bar"}{a:3,s:"baz"}' | - super -z -c 'sample |> sort a' - -``` - -```mdtest-output +```mdtest-spq +# spq +sample | sort a +# input +{a:1} +{a:2} +{s:"foo"} +{s:"bar"} +{a:3,s:"baz"} +# expected output {a:1} {a:3,s:"baz"} {s:"foo"} diff --git a/docs/language/operators/search.md b/docs/language/operators/search.md index 69aeef6b66..e3b07a5606 100644 --- a/docs/language/operators/search.md +++ b/docs/language/operators/search.md @@ -22,80 +22,114 @@ it is good practice to include the optional keyword. ### Examples _A simple keyword search for "world"_ -```mdtest-command -echo '"hello, world" "say hello" "goodbye, world"' | super -z -c 'search world' - -``` - -```mdtest-output +```mdtest-spq +# spq +search world +# input +"hello, world" +"say hello" +"goodbye, world" +# expected output "hello, world" "goodbye, world" -``` -Search can utilize _arithmetic comparisons_ -```mdtest-command -echo '1 2 3' | super -z -c 'search this >= 2' - ``` -```mdtest-output +Search can utilize _arithmetic comparisons_ +```mdtest-spq +# spq +search this >= 2 +# input +1 +2 +3 +# expected output 2 3 -``` -_The "search" keyword may be dropped_ -```mdtest-command -echo '1 2 3' | super -z -c '? 2 or 3' - ``` -```mdtest-output +_The "search" keyword may be dropped_ +```mdtest-spq +# spq +? 2 or 3 +# input +1 +2 +3 +# expected output 2 3 -``` -_A search with [Boolean logic](../search-expressions.md#boolean-logic)_ -```mdtest-command -echo '1 2 3' | super -z -c 'search this >= 2 AND this <= 2' - ``` -```mdtest-output +_A search with [Boolean logic](../search-expressions.md#boolean-logic)_ +```mdtest-spq +# spq +search this >= 2 AND this <= 2 +# input +1 +2 +3 +# expected output 2 -``` -_The AND operator may be omitted through predicate concatenation_ -```mdtest-command -echo '1 2 3' | super -z -c 'search this >= 2 this <= 2' - ``` -```mdtest-output +_The AND operator may be omitted through predicate concatenation_ +```mdtest-spq +# spq +search this >= 2 this <= 2 +# input +1 2 3 +# expected output 2 ``` -_Concatenation for keyword search_ -```mdtest-command -echo '"foo" "foo bar" "foo bar baz" "baz"' | super -z -c '? foo bar' - -``` -```mdtest-output +_Concatenation for keyword search_ +```mdtest-spq +# spq +? foo bar +# input +"foo" +"foo bar" +"foo bar baz" +"baz" +# expected output "foo bar" "foo bar baz" -``` -_Search expressions match fields names too_ -```mdtest-command -echo '{foo:1} {bar:2} {foo:3}' | super -z -c '? foo' - ``` -```mdtest-output +_Search expressions match fields names too_ +```mdtest-spq +# spq +? foo +# input +{foo:1} +{bar:2} +{foo:3} +# expected output {foo:1} {foo:3} -``` -_Boolean functions may be called_ -```mdtest-command -echo '1 "foo" 10.0.0.1' | super -z -c 'search is()' - ``` -```mdtest-output +_Boolean functions may be called_ +```mdtest-spq +# spq +search is() +# input +1 +"foo" +10.0.0.1 +# expected output 1 -``` -_Boolean functions with Boolean logic_ -```mdtest-command -echo '1 "foo" 10.0.0.1' | super -z -c 'search is() or is()' - ``` -```mdtest-output +_Boolean functions with Boolean logic_ +```mdtest-spq +# spq +search is() or is() +# input +1 +"foo" +10.0.0.1 +# expected output 1 10.0.0.1 ```