Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Go: database local source models #17905

Open
wants to merge 31 commits into
base: main
Choose a base branch
from

Conversation

egregius313
Copy link
Contributor

@egregius313 egregius313 commented Nov 4, 2024

Pull Request checklist

All query authors

Internal query authors only

  • Autofixes generated based on these changes are valid, only needed if this PR makes significant changes to .ql, .qll, or .qhelp files. See the documentation (internal access required).
  • Changes are validated at scale (internal access required).
  • Adding a new query? Consider also adding the query to autofix.

@github-actions github-actions bot added the Go label Nov 4, 2024
@egregius313 egregius313 changed the title Egregius313/go/dataflow/models/database Go: database local source models Nov 4, 2024
Copy link
Contributor

github-actions bot commented Nov 4, 2024

⚠️ The head of this PR and the base branch were compared for differences in the framework coverage reports. The generated reports are available in the artifacts of this workflow run. The differences will be picked up by the nightly job after the PR gets merged.

Click to show differences in coverage

go

Generated file changes for go

  • Changes to framework-coverage-go.rst:
-    `Couchbase official client(gocb) <https://github.com/couchbase/gocb>`_,"``github.com/couchbase/gocb*``, ``gopkg.in/couchbase/gocb*``",,36,16
+    `Couchbase official client(gocb) <https://github.com/couchbase/gocb>`_,"``github.com/couchbase/gocb*``, ``gopkg.in/couchbase/gocb*``",26,56,16
-    `Couchbase unofficial client <http://www.github.com/couchbase/go-couchbase>`_,``github.com/couchbaselabs/gocb*``,,18,8
+    `Couchbase unofficial client <http://www.github.com/couchbase/go-couchbase>`_,``github.com/couchbaselabs/gocb*``,13,28,8
-    `Standard library <https://pkg.go.dev/std>`_,"````, ``archive/*``, ``bufio``, ``bytes``, ``cmp``, ``compress/*``, ``container/*``, ``context``, ``crypto``, ``crypto/*``, ``database/*``, ``debug/*``, ``embed``, ``encoding``, ``encoding/*``, ``errors``, ``expvar``, ``flag``, ``fmt``, ``go/*``, ``hash``, ``hash/*``, ``html``, ``html/*``, ``image``, ``image/*``, ``index/*``, ``io``, ``io/*``, ``log``, ``log/*``, ``maps``, ``math``, ``math/*``, ``mime``, ``mime/*``, ``net``, ``net/*``, ``os``, ``os/*``, ``path``, ``path/*``, ``plugin``, ``reflect``, ``reflect/*``, ``regexp``, ``regexp/*``, ``slices``, ``sort``, ``strconv``, ``strings``, ``sync``, ``sync/*``, ``syscall``, ``syscall/*``, ``testing``, ``testing/*``, ``text/*``, ``time``, ``time/*``, ``unicode``, ``unicode/*``, ``unsafe``",33,604,104
+    `Standard library <https://pkg.go.dev/std>`_,"````, ``archive/*``, ``bufio``, ``bytes``, ``cmp``, ``compress/*``, ``container/*``, ``context``, ``crypto``, ``crypto/*``, ``database/*``, ``debug/*``, ``embed``, ``encoding``, ``encoding/*``, ``errors``, ``expvar``, ``flag``, ``fmt``, ``go/*``, ``hash``, ``hash/*``, ``html``, ``html/*``, ``image``, ``image/*``, ``index/*``, ``io``, ``io/*``, ``log``, ``log/*``, ``maps``, ``math``, ``math/*``, ``mime``, ``mime/*``, ``net``, ``net/*``, ``os``, ``os/*``, ``path``, ``path/*``, ``plugin``, ``reflect``, ``reflect/*``, ``regexp``, ``regexp/*``, ``slices``, ``sort``, ``strconv``, ``strings``, ``sync``, ``sync/*``, ``syscall``, ``syscall/*``, ``testing``, ``testing/*``, ``text/*``, ``time``, ``time/*``, ``unicode``, ``unicode/*``, ``unsafe``",51,607,104
-    `beego <https://beego.me/>`_,"``github.com/astaxie/beego*``, ``github.com/beego/beego*``",63,63,213
+    `beego <https://beego.me/>`_,"``github.com/astaxie/beego*``, ``github.com/beego/beego*``",93,63,213
-    Others,"``github.com/Masterminds/squirrel``, ``github.com/caarlos0/env``, ``github.com/go-gorm/gorm``, ``github.com/go-xorm/xorm``, ``github.com/gobuffalo/envy``, ``github.com/gogf/gf/database/gdb``, ``github.com/hashicorp/go-envparse``, ``github.com/jinzhu/gorm``, ``github.com/jmoiron/sqlx``, ``github.com/joho/godotenv``, ``github.com/kelseyhightower/envconfig``, ``github.com/lann/squirrel``, ``github.com/raindog308/gorqlite``, ``github.com/rqlite/gorqlite``, ``github.com/uptrace/bun``, ``go.mongodb.org/mongo-driver/mongo``, ``gopkg.in/Masterminds/squirrel``, ``gorm.io/gorm``, ``xorm.io/xorm``",23,2,391
+    Others,"``github.com/Masterminds/squirrel``, ``github.com/caarlos0/env``, ``github.com/go-gorm/gorm``, ``github.com/go-xorm/xorm``, ``github.com/gobuffalo/envy``, ``github.com/gogf/gf/database/gdb``, ``github.com/hashicorp/go-envparse``, ``github.com/jinzhu/gorm``, ``github.com/jmoiron/sqlx``, ``github.com/joho/godotenv``, ``github.com/kelseyhightower/envconfig``, ``github.com/lann/squirrel``, ``github.com/raindog308/gorqlite``, ``github.com/rqlite/gorqlite``, ``github.com/uptrace/bun``, ``go.mongodb.org/mongo-driver/mongo``, ``gopkg.in/Masterminds/squirrel``, ``gorm.io/gorm``, ``xorm.io/xorm``",179,56,391
-    Totals,,307,928,1532
+    Totals,,550,1015,1532
  • Changes to framework-coverage-go.csv:
- package,sink,source,summary,sink:command-injection,sink:credentials-key,sink:jwt,sink:log-injection,sink:nosql-injection,sink:path-injection,sink:regex-use[0],sink:regex-use[1],sink:regex-use[c],sink:request-forgery,sink:request-forgery[TCP Addr + Port],sink:sql-injection,sink:url-redirection,sink:url-redirection[0],sink:url-redirection[receiver],sink:xpath-injection,source:environment,source:file,source:remote,source:stdin,summary:taint,summary:value
+ package,sink,source,summary,sink:command-injection,sink:credentials-key,sink:jwt,sink:log-injection,sink:nosql-injection,sink:path-injection,sink:regex-use[0],sink:regex-use[1],sink:regex-use[c],sink:request-forgery,sink:request-forgery[TCP Addr + Port],sink:sql-injection,sink:url-redirection,sink:url-redirection[0],sink:url-redirection[receiver],sink:xpath-injection,source:database,source:environment,source:file,source:remote,source:stdin,summary:taint,summary:value
- ,,,8,,,,,,,,,,,,,,,,,,,,,3,5
+ ,,,8,,,,,,,,,,,,,,,,,,,,,,3,5
- archive/tar,,,5,,,,,,,,,,,,,,,,,,,,,5,
+ archive/tar,,,5,,,,,,,,,,,,,,,,,,,,,,5,
- archive/zip,,,6,,,,,,,,,,,,,,,,,,,,,6,
+ archive/zip,,,6,,,,,,,,,,,,,,,,,,,,,,6,
- bufio,,,17,,,,,,,,,,,,,,,,,,,,,17,
+ bufio,,,17,,,,,,,,,,,,,,,,,,,,,,17,
- bytes,,,43,,,,,,,,,,,,,,,,,,,,,43,
+ bytes,,,43,,,,,,,,,,,,,,,,,,,,,,43,
- clevergo.tech/clevergo,1,,,,,,,,,,,,,,,,,1,,,,,,,
+ clevergo.tech/clevergo,1,,,,,,,,,,,,,,,,,1,,,,,,,,
- compress/bzip2,,,1,,,,,,,,,,,,,,,,,,,,,1,
+ compress/bzip2,,,1,,,,,,,,,,,,,,,,,,,,,,1,
- compress/flate,,,4,,,,,,,,,,,,,,,,,,,,,4,
+ compress/flate,,,4,,,,,,,,,,,,,,,,,,,,,,4,
- compress/gzip,,,3,,,,,,,,,,,,,,,,,,,,,3,
+ compress/gzip,,,3,,,,,,,,,,,,,,,,,,,,,,3,
- compress/lzw,,,1,,,,,,,,,,,,,,,,,,,,,1,
+ compress/lzw,,,1,,,,,,,,,,,,,,,,,,,,,,1,
- compress/zlib,,,4,,,,,,,,,,,,,,,,,,,,,4,
+ compress/zlib,,,4,,,,,,,,,,,,,,,,,,,,,,4,
- container/heap,,,5,,,,,,,,,,,,,,,,,,,,,5,
+ container/heap,,,5,,,,,,,,,,,,,,,,,,,,,,5,
- container/list,,,20,,,,,,,,,,,,,,,,,,,,,20,
+ container/list,,,20,,,,,,,,,,,,,,,,,,,,,,20,
- container/ring,,,5,,,,,,,,,,,,,,,,,,,,,5,
+ container/ring,,,5,,,,,,,,,,,,,,,,,,,,,,5,
- context,,,5,,,,,,,,,,,,,,,,,,,,,5,
+ context,,,5,,,,,,,,,,,,,,,,,,,,,,5,
- crypto,,,10,,,,,,,,,,,,,,,,,,,,,10,
+ crypto,,,10,,,,,,,,,,,,,,,,,,,,,,10,
- database/sql,30,,11,,,,,,,,,,,,30,,,,,,,,,11,
+ database/sql,30,18,14,,,,,,,,,,,,30,,,,,18,,,,,14,
- encoding,,,77,,,,,,,,,,,,,,,,,,,,,77,
+ encoding,,,77,,,,,,,,,,,,,,,,,,,,,,77,
- errors,,,3,,,,,,,,,,,,,,,,,,,,,3,
+ errors,,,3,,,,,,,,,,,,,,,,,,,,,,3,
- expvar,,,6,,,,,,,,,,,,,,,,,,,,,6,
+ expvar,,,6,,,,,,,,,,,,,,,,,,,,,,6,
- fmt,3,,16,,,,3,,,,,,,,,,,,,,,,,16,
+ fmt,3,,16,,,,3,,,,,,,,,,,,,,,,,,16,
- github.com/ChrisTrenkamp/goxpath,3,,,,,,,,,,,,,,,,,,3,,,,,,
+ github.com/ChrisTrenkamp/goxpath,3,,,,,,,,,,,,,,,,,,3,,,,,,,
- github.com/Masterminds/squirrel,32,,,,,,,,,,,,,,32,,,,,,,,,,
+ github.com/Masterminds/squirrel,32,8,1,,,,,,,,,,,,32,,,,,8,,,,,1,
- github.com/Sirupsen/logrus,145,,,,,,145,,,,,,,,,,,,,,,,,,
+ github.com/Sirupsen/logrus,145,,,,,,145,,,,,,,,,,,,,,,,,,,
- github.com/antchfx/htmlquery,4,,,,,,,,,,,,,,,,,,4,,,,,,
+ github.com/antchfx/htmlquery,4,,,,,,,,,,,,,,,,,,4,,,,,,,
- github.com/antchfx/jsonquery,4,,,,,,,,,,,,,,,,,,4,,,,,,
+ github.com/antchfx/jsonquery,4,,,,,,,,,,,,,,,,,,4,,,,,,,
- github.com/antchfx/xmlquery,8,,,,,,,,,,,,,,,,,,8,,,,,,
+ github.com/antchfx/xmlquery,8,,,,,,,,,,,,,,,,,,8,,,,,,,
- github.com/antchfx/xpath,4,,,,,,,,,,,,,,,,,,4,,,,,,
+ github.com/antchfx/xpath,4,,,,,,,,,,,,,,,,,,4,,,,,,,
- github.com/appleboy/gin-jwt,1,,,,1,,,,,,,,,,,,,,,,,,,,
+ github.com/appleboy/gin-jwt,1,,,,1,,,,,,,,,,,,,,,,,,,,,
- github.com/astaxie/beego,71,21,21,,,,34,,5,,,,,,30,2,,,,,,21,,21,
+ github.com/astaxie/beego,71,31,21,,,,34,,5,,,,,,30,2,,,,10,,,21,,21,
- github.com/beego/beego,142,42,42,,,,68,,10,,,,,,60,4,,,,,,42,,42,
+ github.com/beego/beego,142,62,42,,,,68,,10,,,,,,60,4,,,,20,,,42,,42,
- github.com/caarlos0/env,,5,2,,,,,,,,,,,,,,,,,5,,,,1,1
+ github.com/caarlos0/env,,5,2,,,,,,,,,,,,,,,,,,5,,,,1,1
- github.com/clevergo/clevergo,1,,,,,,,,,,,,,,,,,1,,,,,,,
+ github.com/clevergo/clevergo,1,,,,,,,,,,,,,,,,,1,,,,,,,,
- github.com/codeskyblue/go-sh,4,,,4,,,,,,,,,,,,,,,,,,,,,
+ github.com/codeskyblue/go-sh,4,,,4,,,,,,,,,,,,,,,,,,,,,,
- github.com/couchbase/gocb,8,,18,,,,,8,,,,,,,,,,,,,,,,18,
+ github.com/couchbase/gocb,8,13,28,,,,,8,,,,,,,,,,,,13,,,,,28,
- github.com/couchbaselabs/gocb,8,,18,,,,,8,,,,,,,,,,,,,,,,18,
+ github.com/couchbaselabs/gocb,8,13,28,,,,,8,,,,,,,,,,,,13,,,,,28,
- github.com/crankycoder/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,
+ github.com/crankycoder/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,,
- github.com/cristalhq/jwt,1,,,,1,,,,,,,,,,,,,,,,,,,,
+ github.com/cristalhq/jwt,1,,,,1,,,,,,,,,,,,,,,,,,,,,
- github.com/davecgh/go-spew/spew,9,,,,,,9,,,,,,,,,,,,,,,,,,
+ github.com/davecgh/go-spew/spew,9,,,,,,9,,,,,,,,,,,,,,,,,,,
- github.com/dgrijalva/jwt-go,3,,9,,2,1,,,,,,,,,,,,,,,,,,9,
+ github.com/dgrijalva/jwt-go,3,,9,,2,1,,,,,,,,,,,,,,,,,,,9,
- github.com/elazarl/goproxy,2,2,2,,,,2,,,,,,,,,,,,,,,2,,2,
+ github.com/elazarl/goproxy,2,2,2,,,,2,,,,,,,,,,,,,,,,2,,2,
- github.com/emicklei/go-restful,,7,,,,,,,,,,,,,,,,,,,,7,,,
+ github.com/emicklei/go-restful,,7,,,,,,,,,,,,,,,,,,,,,7,,,
- github.com/evanphx/json-patch,,,12,,,,,,,,,,,,,,,,,,,,,12,
+ github.com/evanphx/json-patch,,,12,,,,,,,,,,,,,,,,,,,,,,12,
- github.com/form3tech-oss/jwt-go,2,,,,2,,,,,,,,,,,,,,,,,,,,
+ github.com/form3tech-oss/jwt-go,2,,,,2,,,,,,,,,,,,,,,,,,,,,
- github.com/gin-gonic/gin,3,46,2,,,,,,3,,,,,,,,,,,,,46,,2,
+ github.com/gin-gonic/gin,3,46,2,,,,,,3,,,,,,,,,,,,,,46,,2,
- github.com/go-chi/chi,,3,,,,,,,,,,,,,,,,,,,,3,,,
+ github.com/go-chi/chi,,3,,,,,,,,,,,,,,,,,,,,,3,,,
- github.com/go-chi/jwtauth,1,,,,1,,,,,,,,,,,,,,,,,,,,
+ github.com/go-chi/jwtauth,1,,,,1,,,,,,,,,,,,,,,,,,,,,
- github.com/go-gorm/gorm,13,,,,,,,,,,,,,,13,,,,,,,,,,
+ github.com/go-gorm/gorm,13,8,,,,,,,,,,,,,13,,,,,8,,,,,,
- github.com/go-jose/go-jose,3,,4,,2,1,,,,,,,,,,,,,,,,,,4,
+ github.com/go-jose/go-jose,3,,4,,2,1,,,,,,,,,,,,,,,,,,,4,
- github.com/go-kit/kit/auth/jwt,1,,,,1,,,,,,,,,,,,,,,,,,,,
+ github.com/go-kit/kit/auth/jwt,1,,,,1,,,,,,,,,,,,,,,,,,,,,
- github.com/go-pg/pg/orm,,,6,,,,,,,,,,,,,,,,,,,,,6,
+ github.com/go-pg/pg/orm,,,6,,,,,,,,,,,,,,,,,,,,,,6,
- github.com/go-xmlpath/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,
+ github.com/go-xmlpath/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,,
- github.com/go-xorm/xorm,34,,,,,,,,,,,,,,34,,,,,,,,,,
+ github.com/go-xorm/xorm,34,,,,,,,,,,,,,,34,,,,,,,,,,,
- github.com/gobuffalo/envy,,7,,,,,,,,,,,,,,,,,,7,,,,,
+ github.com/gobuffalo/envy,,7,,,,,,,,,,,,,,,,,,,7,,,,,
- github.com/gobwas/ws,,2,,,,,,,,,,,,,,,,,,,,2,,,
+ github.com/gobwas/ws,,2,,,,,,,,,,,,,,,,,,,,,2,,,
- github.com/gofiber/fiber,5,,,,,,,,4,,,,,,,,,1,,,,,,,
+ github.com/gofiber/fiber,5,,,,,,,,4,,,,,,,,,1,,,,,,,,
- github.com/gogf/gf-jwt,1,,,,1,,,,,,,,,,,,,,,,,,,,
+ github.com/gogf/gf-jwt,1,,,,1,,,,,,,,,,,,,,,,,,,,,
- github.com/gogf/gf/database/gdb,51,,,,,,,,,,,,,,51,,,,,,,,,,
+ github.com/gogf/gf/database/gdb,51,12,30,,,,,,,,,,,,51,,,,,12,,,,,30,
- github.com/going/toolkit/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,
+ github.com/going/toolkit/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,,
- github.com/golang-jwt/jwt,3,,11,,2,1,,,,,,,,,,,,,,,,,,11,
+ github.com/golang-jwt/jwt,3,,11,,2,1,,,,,,,,,,,,,,,,,,,11,
- github.com/golang/glog,90,,,,,,90,,,,,,,,,,,,,,,,,,
+ github.com/golang/glog,90,,,,,,90,,,,,,,,,,,,,,,,,,,
- github.com/golang/protobuf/proto,,,4,,,,,,,,,,,,,,,,,,,,,4,
+ github.com/golang/protobuf/proto,,,4,,,,,,,,,,,,,,,,,,,,,,4,
- github.com/gorilla/mux,,1,,,,,,,,,,,,,,,,,,,,1,,,
+ github.com/gorilla/mux,,1,,,,,,,,,,,,,,,,,,,,,1,,,
- github.com/gorilla/websocket,,3,,,,,,,,,,,,,,,,,,,,3,,,
+ github.com/gorilla/websocket,,3,,,,,,,,,,,,,,,,,,,,,3,,,
- github.com/hashicorp/go-envparse,,1,,,,,,,,,,,,,,,,,,1,,,,,
+ github.com/hashicorp/go-envparse,,1,,,,,,,,,,,,,,,,,,,1,,,,,
- github.com/jbowtie/gokogiri/xml,4,,,,,,,,,,,,,,,,,,4,,,,,,
+ github.com/jbowtie/gokogiri/xml,4,,,,,,,,,,,,,,,,,,4,,,,,,,
- github.com/jbowtie/gokogiri/xpath,1,,,,,,,,,,,,,,,,,,1,,,,,,
+ github.com/jbowtie/gokogiri/xpath,1,,,,,,,,,,,,,,,,,,1,,,,,,,
- github.com/jinzhu/gorm,13,,,,,,,,,,,,,,13,,,,,,,,,,
+ github.com/jinzhu/gorm,13,8,,,,,,,,,,,,,13,,,,,8,,,,,,
- github.com/jmoiron/sqlx,12,,,,,,,,,,,,,,12,,,,,,,,,,
+ github.com/jmoiron/sqlx,12,55,12,,,,,,,,,,,,12,,,,,55,,,,,12,
- github.com/joho/godotenv,,4,,,,,,,,,,,,,,,,,,4,,,,,
+ github.com/joho/godotenv,,4,,,,,,,,,,,,,,,,,,,4,,,,,
- github.com/json-iterator/go,,,4,,,,,,,,,,,,,,,,,,,,,4,
+ github.com/json-iterator/go,,,4,,,,,,,,,,,,,,,,,,,,,,4,
- github.com/kataras/iris/context,6,,,,,,,,6,,,,,,,,,,,,,,,,
+ github.com/kataras/iris/context,6,,,,,,,,6,,,,,,,,,,,,,,,,,
- github.com/kataras/iris/middleware/jwt,2,,,,2,,,,,,,,,,,,,,,,,,,,
+ github.com/kataras/iris/middleware/jwt,2,,,,2,,,,,,,,,,,,,,,,,,,,,
- github.com/kataras/iris/server/web/context,6,,,,,,,,6,,,,,,,,,,,,,,,,
+ github.com/kataras/iris/server/web/context,6,,,,,,,,6,,,,,,,,,,,,,,,,,
- github.com/kataras/jwt,5,,,,5,,,,,,,,,,,,,,,,,,,,
+ github.com/kataras/jwt,5,,,,5,,,,,,,,,,,,,,,,,,,,,
- github.com/kelseyhightower/envconfig,,6,,,,,,,,,,,,,,,,,,6,,,,,
+ github.com/kelseyhightower/envconfig,,6,,,,,,,,,,,,,,,,,,,6,,,,,
- github.com/labstack/echo,3,12,2,,,,,,2,,,,,,,1,,,,,,12,,2,
+ github.com/labstack/echo,3,12,2,,,,,,2,,,,,,,1,,,,,,,12,,2,
- github.com/lann/squirrel,32,,,,,,,,,,,,,,32,,,,,,,,,,
+ github.com/lann/squirrel,32,8,1,,,,,,,,,,,,32,,,,,8,,,,,1,
- github.com/lestrrat-go/jwx,2,,,,2,,,,,,,,,,,,,,,,,,,,
+ github.com/lestrrat-go/jwx,2,,,,2,,,,,,,,,,,,,,,,,,,,,
- github.com/lestrrat-go/libxml2/parser,3,,,,,,,,,,,,,,,,,,3,,,,,,
+ github.com/lestrrat-go/libxml2/parser,3,,,,,,,,,,,,,,,,,,3,,,,,,,
- github.com/lestrrat/go-jwx/jwk,1,,,,1,,,,,,,,,,,,,,,,,,,,
+ github.com/lestrrat/go-jwx/jwk,1,,,,1,,,,,,,,,,,,,,,,,,,,,
- github.com/masterzen/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,
+ github.com/masterzen/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,,
- github.com/moovweb/gokogiri/xml,4,,,,,,,,,,,,,,,,,,4,,,,,,
+ github.com/moovweb/gokogiri/xml,4,,,,,,,,,,,,,,,,,,4,,,,,,,
- github.com/moovweb/gokogiri/xpath,1,,,,,,,,,,,,,,,,,,1,,,,,,
+ github.com/moovweb/gokogiri/xpath,1,,,,,,,,,,,,,,,,,,1,,,,,,,
- github.com/ory/fosite/token/jwt,2,,,,2,,,,,,,,,,,,,,,,,,,,
+ github.com/ory/fosite/token/jwt,2,,,,2,,,,,,,,,,,,,,,,,,,,,
- github.com/raindog308/gorqlite,24,,,,,,,,,,,,,,24,,,,,,,,,,
+ github.com/raindog308/gorqlite,24,8,2,,,,,,,,,,,,24,,,,,8,,,,,2,
- github.com/revel/revel,2,23,10,,,,,,1,,,,,,,1,,,,,,23,,10,
+ github.com/revel/revel,2,23,10,,,,,,1,,,,,,,1,,,,,,,23,,10,
- github.com/robfig/revel,2,23,10,,,,,,1,,,,,,,1,,,,,,23,,10,
+ github.com/robfig/revel,2,23,10,,,,,,1,,,,,,,1,,,,,,,23,,10,
- github.com/rqlite/gorqlite,24,,,,,,,,,,,,,,24,,,,,,,,,,
+ github.com/rqlite/gorqlite,24,8,2,,,,,,,,,,,,24,,,,,8,,,,,2,
- github.com/santhosh-tekuri/xpathparser,2,,,,,,,,,,,,,,,,,,2,,,,,,
+ github.com/santhosh-tekuri/xpathparser,2,,,,,,,,,,,,,,,,,,2,,,,,,,
- github.com/sendgrid/sendgrid-go/helpers/mail,,,1,,,,,,,,,,,,,,,,,,,,,1,
+ github.com/sendgrid/sendgrid-go/helpers/mail,,,1,,,,,,,,,,,,,,,,,,,,,,1,
- github.com/sirupsen/logrus,145,,,,,,145,,,,,,,,,,,,,,,,,,
+ github.com/sirupsen/logrus,145,,,,,,145,,,,,,,,,,,,,,,,,,,
- github.com/spf13/afero,34,,,,,,,,34,,,,,,,,,,,,,,,,
+ github.com/spf13/afero,34,,,,,,,,34,,,,,,,,,,,,,,,,,
- github.com/square/go-jose,3,,4,,2,1,,,,,,,,,,,,,,,,,,4,
+ github.com/square/go-jose,3,,4,,2,1,,,,,,,,,,,,,,,,,,,4,
- github.com/uptrace/bun,63,,,,,,,,,,,,,,63,,,,,,,,,,
+ github.com/uptrace/bun,63,18,,,,,,,,,,,,,63,,,,,18,,,,,,
- github.com/valyala/fasthttp,35,50,5,,,,,,8,,,,17,8,,2,,,,,,50,,5,
+ github.com/valyala/fasthttp,35,50,5,,,,,,8,,,,17,8,,2,,,,,,,50,,5,
- go.mongodb.org/mongo-driver/mongo,14,,,,,,,14,,,,,,,,,,,,,,,,,
+ go.mongodb.org/mongo-driver/mongo,14,7,5,,,,,14,,,,,,,,,,,,7,,,,,5,
- go.uber.org/zap,33,,11,,,,33,,,,,,,,,,,,,,,,,11,
+ go.uber.org/zap,33,,11,,,,33,,,,,,,,,,,,,,,,,,11,
- golang.org/x/crypto/ssh,4,,,4,,,,,,,,,,,,,,,,,,,,,
+ golang.org/x/crypto/ssh,4,,,4,,,,,,,,,,,,,,,,,,,,,,
- golang.org/x/net/context,,,5,,,,,,,,,,,,,,,,,,,,,5,
+ golang.org/x/net/context,,,5,,,,,,,,,,,,,,,,,,,,,,5,
- golang.org/x/net/html,,,16,,,,,,,,,,,,,,,,,,,,,16,
+ golang.org/x/net/html,,,16,,,,,,,,,,,,,,,,,,,,,,16,
- golang.org/x/net/websocket,,2,,,,,,,,,,,,,,,,,,,,2,,,
+ golang.org/x/net/websocket,,2,,,,,,,,,,,,,,,,,,,,,2,,,
- google.golang.org/protobuf/internal/encoding/text,,,1,,,,,,,,,,,,,,,,,,,,,1,
+ google.golang.org/protobuf/internal/encoding/text,,,1,,,,,,,,,,,,,,,,,,,,,,1,
- google.golang.org/protobuf/internal/impl,,,2,,,,,,,,,,,,,,,,,,,,,2,
+ google.golang.org/protobuf/internal/impl,,,2,,,,,,,,,,,,,,,,,,,,,,2,
- google.golang.org/protobuf/proto,,,8,,,,,,,,,,,,,,,,,,,,,8,
+ google.golang.org/protobuf/proto,,,8,,,,,,,,,,,,,,,,,,,,,,8,
- google.golang.org/protobuf/reflect/protoreflect,,,1,,,,,,,,,,,,,,,,,,,,,1,
+ google.golang.org/protobuf/reflect/protoreflect,,,1,,,,,,,,,,,,,,,,,,,,,,1,
- gopkg.in/Masterminds/squirrel,32,,,,,,,,,,,,,,32,,,,,,,,,,
+ gopkg.in/Masterminds/squirrel,32,8,1,,,,,,,,,,,,32,,,,,8,,,,,1,
- gopkg.in/couchbase/gocb,8,,18,,,,,8,,,,,,,,,,,,,,,,18,
+ gopkg.in/couchbase/gocb,8,13,28,,,,,8,,,,,,,,,,,,13,,,,,28,
- gopkg.in/glog,90,,,,,,90,,,,,,,,,,,,,,,,,,
+ gopkg.in/glog,90,,,,,,90,,,,,,,,,,,,,,,,,,,
- gopkg.in/go-jose/go-jose,3,,4,,2,1,,,,,,,,,,,,,,,,,,4,
+ gopkg.in/go-jose/go-jose,3,,4,,2,1,,,,,,,,,,,,,,,,,,,4,
- gopkg.in/go-xmlpath/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,
+ gopkg.in/go-xmlpath/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,,
- gopkg.in/macaron,1,12,1,,,,,,,,,,,,,,,1,,,,12,,1,
+ gopkg.in/macaron,1,12,1,,,,,,,,,,,,,,,1,,,,,12,,1,
- gopkg.in/square/go-jose,3,,4,,2,1,,,,,,,,,,,,,,,,,,4,
+ gopkg.in/square/go-jose,3,,4,,2,1,,,,,,,,,,,,,,,,,,,4,
- gopkg.in/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,
+ gopkg.in/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,,
- gopkg.in/yaml,,,9,,,,,,,,,,,,,,,,,,,,,9,
+ gopkg.in/yaml,,,9,,,,,,,,,,,,,,,,,,,,,,9,
- gorm.io/gorm,13,,,,,,,,,,,,,,13,,,,,,,,,,
+ gorm.io/gorm,13,8,,,,,,,,,,,,,13,,,,,8,,,,,,
- html,,,8,,,,,,,,,,,,,,,,,,,,,8,
+ html,,,8,,,,,,,,,,,,,,,,,,,,,,8,
- io,5,4,34,,,,,,5,,,,,,,,,,,,4,,,34,
+ io,5,4,34,,,,,,5,,,,,,,,,,,,,4,,,34,
- k8s.io/api/core,,,10,,,,,,,,,,,,,,,,,,,,,10,
+ k8s.io/api/core,,,10,,,,,,,,,,,,,,,,,,,,,,10,
- k8s.io/apimachinery/pkg/runtime,,,47,,,,,,,,,,,,,,,,,,,,,47,
+ k8s.io/apimachinery/pkg/runtime,,,47,,,,,,,,,,,,,,,,,,,,,,47,
- k8s.io/klog,90,,,,,,90,,,,,,,,,,,,,,,,,,
+ k8s.io/klog,90,,,,,,90,,,,,,,,,,,,,,,,,,,
- launchpad.net/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,
+ launchpad.net/xmlpath,2,,,,,,,,,,,,,,,,,,2,,,,,,,
- log,20,,3,,,,20,,,,,,,,,,,,,,,,,3,
+ log,20,,3,,,,20,,,,,,,,,,,,,,,,,,3,
- math/big,,,1,,,,,,,,,,,,,,,,,,,,,1,
+ math/big,,,1,,,,,,,,,,,,,,,,,,,,,,1,
- mime,,,14,,,,,,,,,,,,,,,,,,,,,14,
+ mime,,,14,,,,,,,,,,,,,,,,,,,,,,14,
- net,2,16,100,,,,,,1,,,,,,,,1,,,,,16,,100,
+ net,2,16,100,,,,,,1,,,,,,,,1,,,,,,16,,100,
- nhooyr.io/websocket,,2,,,,,,,,,,,,,,,,,,,,2,,,
+ nhooyr.io/websocket,,2,,,,,,,,,,,,,,,,,,,,,2,,,
- os,29,11,6,3,,,,,26,,,,,,,,,,,7,3,,1,6,
+ os,29,11,6,3,,,,,26,,,,,,,,,,,,7,3,,1,6,
- path,,,18,,,,,,,,,,,,,,,,,,,,,18,
+ path,,,18,,,,,,,,,,,,,,,,,,,,,,18,
- reflect,,,37,,,,,,,,,,,,,,,,,,,,,37,
+ reflect,,,37,,,,,,,,,,,,,,,,,,,,,,37,
- regexp,10,,20,,,,,,,3,3,4,,,,,,,,,,,,20,
+ regexp,10,,20,,,,,,,3,3,4,,,,,,,,,,,,,20,
- slices,,,17,,,,,,,,,,,,,,,,,,,,,,17
+ slices,,,17,,,,,,,,,,,,,,,,,,,,,,,17
- sort,,,1,,,,,,,,,,,,,,,,,,,,,1,
+ sort,,,1,,,,,,,,,,,,,,,,,,,,,,1,
- strconv,,,9,,,,,,,,,,,,,,,,,,,,,9,
+ strconv,,,9,,,,,,,,,,,,,,,,,,,,,,9,
- strings,,,34,,,,,,,,,,,,,,,,,,,,,34,
+ strings,,,34,,,,,,,,,,,,,,,,,,,,,,34,
- sync,,,34,,,,,,,,,,,,,,,,,,,,,34,
+ sync,,,34,,,,,,,,,,,,,,,,,,,,,,34,
- syscall,5,2,8,5,,,,,,,,,,,,,,,,2,,,,8,
+ syscall,5,2,8,5,,,,,,,,,,,,,,,,,2,,,,8,
- text/scanner,,,3,,,,,,,,,,,,,,,,,,,,,3,
+ text/scanner,,,3,,,,,,,,,,,,,,,,,,,,,,3,
- text/tabwriter,,,1,,,,,,,,,,,,,,,,,,,,,1,
+ text/tabwriter,,,1,,,,,,,,,,,,,,,,,,,,,,1,
- text/template,,,6,,,,,,,,,,,,,,,,,,,,,6,
+ text/template,,,6,,,,,,,,,,,,,,,,,,,,,,6,
- xorm.io/xorm,34,,,,,,,,,,,,,,34,,,,,,,,,,
+ xorm.io/xorm,34,,,,,,,,,,,,,,34,,,,,,,,,,,

@egregius313 egregius313 marked this pull request as ready for review November 6, 2024 16:24
@egregius313 egregius313 requested a review from a team as a code owner November 6, 2024 16:24
@owen-mc
Copy link
Contributor

owen-mc commented Nov 7, 2024

Two tests need to have their results updated. go/ql/test/query-tests/Security/CWE-079/StoredXss.qlref also needs to have a post-processing query added for pretty-printing models, as in this PR.

I haven't had time to look at the models themselves yet. Are these new models, or are you converting models from QL? If the latter, should we also delete the QL models?

@egregius313 egregius313 force-pushed the egregius313/go/dataflow/models/database branch from 95ebab7 to 006f91b Compare November 22, 2024 21:13
@egregius313
Copy link
Contributor Author

Are these new models, or are you converting models from QL? If the latter, should we also delete the QL models?

Yes these are primarily new models. There weren't many existing database models as far as I could tell.

The one package where I'm unsure about the existing models vs the new ones are the beego-orm models. The discrepancy seems to be from API changes in v2

@egregius313 egregius313 requested a review from owen-mc December 3, 2024 03:29
Copy link
Contributor

@owen-mc owen-mc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've only reviewed half of the new models so far (stdlib, beego, couchbase).

I note that we do have some existing database read models - see the source definition for the stored xss query. We should convert them to MaD, deprecate the QL classes where they are only used for that, and use the database MaD sources for that query (ignoring what the active threat model is, I think).

Extremely minor stylistic comment: within one .model.yml file I prefer to put sources first, sinks second and summaries third. This is because there are generally the fewest sources, the second fewest sinks and the most summaries. It makes it slightly easier to read and see at a glance which extensible predicates are being extended. When summaries go first, it is easy to miss that other predicates are extended below.

go/ql/lib/ext/database.sql.model.yml Outdated Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Please test for all sources defined in each package.
  • That will probably make the file a bit longer - I would split this up into a separate source file for each package, so it is easier to find particular tests.

Comment on lines 51 to 56
- ["group:beego-orm", "DQL", True, "Read", "", "", "Argument[0]", "database", "manual"]
- ["group:beego-orm", "DQL", True, "ReadWithCtx", "", "", "Argument[1]", "database", "manual"]
- ["group:beego-orm", "DQL", True, "ReadForUpdate", "", "", "Argument[0]", "database", "manual"]
- ["group:beego-orm", "DQL", True, "ReadForUpdateWithCtx", "", "", "Argument[1]", "database", "manual"]
- ["group:beego-orm", "DQL", True, "ReadOrCreate", "", "", "Argument[0]", "database", "manual"]
- ["group:beego-orm", "DQL", True, "ReadOrCreateWithCtx", "", "", "Argument[1]", "database", "manual"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These models are fine. It's confusing because this type only exists in "github.com/beego/beego/v2/client/orm". I guess it would be slightly more precise to change the package to "github.com/beego/beego/client/orm" (leaving out the v2 so that the model applies to future versions as well). But I don't see any problem leaving it as it is, since the extra things that it might match don't exist.

Comment on lines +62 to +64
- ["group:gocb2", "QueryResult", True, "One", "", "", "Argument[0]", "database", "manual"]
- ["group:gocb2", "QueryResult", True, "Row", "", "", "Argument[0]", "database", "manual"]
Copy link
Contributor

@owen-mc owen-mc Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make more sense to model Cluster.Query and Scope.Query (which are the only way of getting a QueryResult) as the sources and model the two methods above as summary models. There is a nice symmetry that the same function is an sql-injection sink for one argument and a database source for the return value.

Also, add summary models for QueryResult.Raw and QueryResultRaw.NextBytes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think that Cluster.AnalyticsQuery and Scope.AnalyticsQuery should be sources, and all the summary models for QueryResult should be duplicated for AnalyticsResult.

Comment on lines +64 to +66
- ["group:gocb2", "TransactionQueryResult", True, "One", "", "", "Argument[0]", "database", "manual"]
- ["group:gocb2", "TransactionQueryResult", True, "Row", "", "", "Argument[0]", "database", "manual"]
Copy link
Contributor

@owen-mc owen-mc Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above, I think TransactionAttemptContext.Query should be the source and these should be summary models. (It should probably be an sql-injection sink as well - I've noted this down somewhere else, no need to do it in this PR.)

- ["group:gocb2", "Collection", True, "GetAndLock", "", "", "ReturnValue[0]", "database", "manual"]
- ["group:gocb2", "Collection", True, "GetAndTouch", "", "", "ReturnValue[0]", "database", "manual"]
- ["group:gocb2", "Collection", True, "GetAnyReplica", "", "", "ReturnValue[0]", "database", "manual"]
- ["group:gocb2", "Collection", True, "LookupIn", "", "", "ReturnValue[0]", "database", "manual"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably we should model LookupInAllReplicas and LookupInAnyReplica as sources, and a summary model for LookupInAllReplicasResult.Next.

- ["group:gocb2", "Collection", True, "GetAndTouch", "", "", "ReturnValue[0]", "database", "manual"]
- ["group:gocb2", "Collection", True, "GetAnyReplica", "", "", "ReturnValue[0]", "database", "manual"]
- ["group:gocb2", "Collection", True, "LookupIn", "", "", "ReturnValue[0]", "database", "manual"]
- ["group:gocb2", "Collection", True, "Scan", "", "", "ReturnValue[0]", "database", "manual"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also need a summary model for ScanResultItem.Content

Copy link
Contributor

@owen-mc owen-mc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've done another four libraries. jmoiron/sqlx was such hard going!

go/ql/lib/ext/github.com.gogf.gf.database.gdb.model.yml Outdated Show resolved Hide resolved
go/ql/lib/ext/github.com.gogf.gf.database.gdb.model.yml Outdated Show resolved Hide resolved
go/ql/lib/ext/github.com.gogf.gf.database.gdb.model.yml Outdated Show resolved Hide resolved
go/ql/lib/ext/github.com.gogf.gf.database.gdb.model.yml Outdated Show resolved Hide resolved
- ["github.com/gogf/gf/database/gdb", "Model", True, "FindOne", "", "", "Argument[receiver]", "ReturnValue[0]", "taint", "manual"]
- ["github.com/gogf/gf/database/gdb", "Model", True, "FindValue", "", "", "Argument[receiver]", "ReturnValue[0]", "taint", "manual"]
- ["github.com/gogf/gf/database/gdb", "Model", True, "FindScan", "", "", "Argument[receiver]", "Argument[0]", "taint", "manual"]
- ["github.com/gogf/gf/database/gdb", "Model", True, "One", "", "", "Argument[receiver]", "ReturnValue[0]", "taint", "manual"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also want models for methods Scan, ScanList, Select, Struct and Structs on Model.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, most of these methods on Model return a Model, which we aren't modelling at the moment. It would mean a huge number of models though. But without that, the examples they give, like db.Model("user").Where("id", 1).Scan(&user) , won't get the taint tracked through it properly. I'm not sure what to do here, to be honest.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change filename to github.com.rqlite.gorqlite.model.yml

extensible: summaryModel
data:
- ["group:gorqlite", "QueryResult", True, "Map", "", "", "Argument[receiver]", "ReturnValue[0]", "taint", "manual"]
- ["group:gorqlite", "QueryResult", True, "Scan", "", "", "Argument[receiver]", "Argument[0]", "taint", "manual"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remember correctly, this model won't work because dataflow out of a variadic parameter doesn't work currently. It can be modeled in QL using TaintTracking::FunctionModel.

extensible: summaryModel
data:
- ["group:gorqlite", "QueryResult", True, "Map", "", "", "Argument[receiver]", "ReturnValue[0]", "taint", "manual"]
- ["group:gorqlite", "QueryResult", True, "Scan", "", "", "Argument[receiver]", "Argument[0]", "taint", "manual"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a model for QueryResult.Slice.

pack: codeql/go-all
extensible: summaryModel
data:
- ["group:squirrel", "RowScanner", True, "Scan", "", "", "Argument[receiver]", "Argument[0]", "taint", "manual"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remember correctly, this model won't work because dataflow out of a variadic parameter doesn't work currently. It can be modeled in QL using TaintTracking::FunctionModel.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also needs models for QueryContextWith, QueryWith, QueryRowContextWith, QueryRowWith, Row.Scan (it seems to override the inherited Scan method from embedding RowScanner), *Builder.Query (different method signature to QueryRower.Query, unfortunately), *Builder.QueryContext, *Builder.QueryRow, *Builder.QueryRowContext, *Builder.Scan, *Builder.ScanContext. (Do we care about all the Builders? Delete? I guess safest to model them all...)

Copy link
Contributor

@owen-mc owen-mc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've finished going through all the libraries. Modeling is such hard work! Thanks for doing all this - it must have taken a long time.

Comment on lines +85 to +91
- ["github.com/uptrace/bun", "DeleteQuery", True, "Scan", "", "", "Argument[0]", "database", "manual"]
- ["github.com/uptrace/bun", "InsertQuery", True, "Scan", "", "", "Argument[0]", "database", "manual"]
- ["github.com/uptrace/bun", "MergeQuery", True, "Scan", "", "", "Argument[0]", "database", "manual"]
- ["github.com/uptrace/bun", "RawQuery", True, "Scan", "", "", "Argument[0]", "database", "manual"]
- ["github.com/uptrace/bun", "SelectQuery", True, "Scan", "", "", "Argument[0]", "database", "manual"]
- ["github.com/uptrace/bun", "TruncateQuery", True, "Scan", "", "", "Argument[0]", "database", "manual"]
- ["github.com/uptrace/bun", "UpdateQuery", True, "Scan", "", "", "Argument[0]", "database", "manual"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remember correctly, this model won't work because dataflow out of a variadic parameter doesn't work currently. It can be modeled in QL.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last example here shows that we need to model Exec as well (in QL). And this part of the docs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These docs make me think we need source models for DB.Query* and summary models for DB.ScanRow and DB.ScanRows.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also model Collection.Aggregate and Collection.ChangeStream as sources.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs contain this example:

// if user's primary key is non-blank, will use it as condition, then will only update that user's name to `hello`
db.Model(&user).Update("name", "hello")

so I think we need a source model for DB.Model(). It's often used with an unitialised struct, in which case there's no harm in having it as a source.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should model DB.Rows and DB.Row as sources (guide). And DB.ScanRows as a summary model (guide). And DB.Pluck as a source (guide).

go/ql/lib/ext/database.sql.model.yml Outdated Show resolved Hide resolved
@egregius313 egregius313 force-pushed the egregius313/go/dataflow/models/database branch from 3d94e4d to ae4aabc Compare January 8, 2025 19:32
@egregius313 egregius313 force-pushed the egregius313/go/dataflow/models/database branch from ae4aabc to 6ab815f Compare January 8, 2025 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants