Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FS store #25

Merged
merged 6 commits into from
Aug 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
*.local
config.toml
fs
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,4 +47,11 @@ go run ./cmd/github-actions-manager -config config.toml -loglevel DEBUG

7. Under Github tokens page https://github.com/settings/tokens, generate a Github personal access token (classic) with scope `[workflow, notifications]` (this may be more than strictly necessary). Copy the generated `token` to `token` in `config.toml`.

### FSPath

Currently, persistent configs are being stored in a low-density file storage system under `fs`.
If you want to disable config persistence i.e. reconstruct the list of subscribed repos after every restart:

Under `config.toml` -> `[store]`, change `type` to `"InMemory"`.

8. Test the app **in a public channel** (e.g. #team-bot-sandbox). The app was not designed with direct messages in mind and may not work there.
3 changes: 2 additions & 1 deletion examples/config.dev.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ webhookSecret="secret"
addr="0.0.0.0:8000"

[store]
type="InMemory"
type="FS"
FSPath="fs"

[slack]
disabled=false
Expand Down
8 changes: 6 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ require (
github.com/google/go-github/v45 v45.1.0
github.com/gorilla/mux v1.8.0
github.com/slack-go/slack v0.11.0
github.com/smartystreets/goconvey v1.8.1
go.uber.org/zap v1.21.0
golang.org/x/oauth2 v0.0.0-20220622183110-fd043fe589d2
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
Expand All @@ -22,7 +23,9 @@ require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/fsnotify/fsnotify v1.5.4 // indirect
github.com/gopherjs/gopherjs v1.17.2 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/jtolds/gls v4.20.0+incompatible // indirect
github.com/magiconair/properties v1.8.6 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
Expand All @@ -31,6 +34,7 @@ require (
github.com/prometheus/client_model v0.2.0 // indirect
github.com/prometheus/common v0.32.1 // indirect
github.com/prometheus/procfs v0.7.3 // indirect
github.com/smarty/assertions v1.15.0 // indirect
github.com/spf13/afero v1.8.2 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/subosito/gotenv v1.3.0 // indirect
Expand Down Expand Up @@ -81,7 +85,7 @@ require (
go.uber.org/multierr v1.6.0 // indirect
golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4 // indirect
golang.org/x/net v0.0.0-20220520000938-2e3eb7b945c2 // indirect
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a // indirect
golang.org/x/sys v0.6.0 // indirect
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 // indirect
golang.org/x/text v0.3.7 // indirect
google.golang.org/appengine v1.6.7 // indirect
Expand All @@ -91,7 +95,7 @@ require (
gopkg.in/yaml.v3 v3.0.0 // indirect
k8s.io/klog/v2 v2.60.1 // indirect
k8s.io/kube-openapi v0.0.0-20220328201542-3ee0da9b0b42 // indirect
k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9 // indirect
k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9
sigs.k8s.io/json v0.0.0-20211208200746-9f7c6b3444d2 // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.2.1 // indirect
sigs.k8s.io/yaml v1.2.0 // indirect
Expand Down
12 changes: 10 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -230,6 +230,8 @@ github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/googleapis/google-cloud-go-testing v0.0.0-20200911160855-bcd43fbb19e8/go.mod h1:dvDLG8qkwmyD9a/MJJN3XJcT3xFxOKAvTZGvuZmac9g=
github.com/gopherjs/gopherjs v1.17.2 h1:fQnZVsXk8uxXIStYb0N4bGk7jeyTalG/wsZjQ25dO0g=
github.com/gopherjs/gopherjs v1.17.2/go.mod h1:pRRIvn/QzFLrKfvEz3qUuEhtE/zLCWfreZ6J5gM2i+k=
github.com/gorilla/mux v1.8.0 h1:i40aqfkR1h2SlN9hojwV5ZA91wcXFOvkdNIeFDP5koI=
github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So=
github.com/gorilla/websocket v1.4.2 h1:+/TMaTYc4QFitKJxsQ7Yye35DkWvkdLcvGKqM+x0Ufc=
Expand Down Expand Up @@ -259,6 +261,8 @@ github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnr
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
Expand Down Expand Up @@ -365,6 +369,10 @@ github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6Mwd
github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88=
github.com/slack-go/slack v0.11.0 h1:sBBjQz8LY++6eeWhGJNZpRm5jvLRNnWBFZ/cAq58a6k=
github.com/slack-go/slack v0.11.0/go.mod h1:hlGi5oXA+Gt+yWTPP0plCdRKmjsDxecdHxYQdlMQKOw=
github.com/smarty/assertions v1.15.0 h1:cR//PqUBUiQRakZWqBiFFQ9wb8emQGDb0HeGdqGByCY=
github.com/smarty/assertions v1.15.0/go.mod h1:yABtdzeQs6l1brC900WlRNwj6ZR55d7B+E8C6HtKdec=
github.com/smartystreets/goconvey v1.8.1 h1:qGjIddxOk4grTu9JPOU31tVfq3cNdBlNa5sSznIX1xY=
github.com/smartystreets/goconvey v1.8.1/go.mod h1:+/u4qLyY6x1jReYOp7GOM2FSt8aP9CzCZL03bI28W60=
github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
github.com/spf13/afero v1.8.2 h1:xehSyVa0YnHWsJ49JFljMpg1HX19V6NDZ1fkm1Xznbo=
github.com/spf13/afero v1.8.2/go.mod h1:CtAatgMJh6bJEIs48Ay/FOnkljP3WeGUG0MC1RfAqwo=
Expand Down Expand Up @@ -590,8 +598,8 @@ golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a h1:dGzPydgVsqGcTRVwiLJ1jVbufYwmzD3LfVPLKsKg+0k=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 h1:JGgROgKl9N8DuW20oFS5gxc+lE67/N3FcwmBPMe7ArY=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
Expand Down
7 changes: 6 additions & 1 deletion pkg/kv/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,17 +9,22 @@ import (
type Type string

const (
TypeFS Type = "FS"
TypeInMemory Type = "InMemory"
TypeKubeConfigMap Type = "KubeConfigMap"
)

type Config struct {
Type Type `validate:"required,oneof=InMemory KubeConfigMap"`
Type Type `validate:"required,oneof=InMemory KubeConfigMap FS"`
KubeNamespace string `validate:"required_if=Type KubeConfigMap"`
FSPath string `validate:"required_if=Type FS"`
}

func NewStore(logger *zap.Logger, config *Config) (Store, error) {
switch config.Type {
case TypeFS:
return NewFSStore(logger, config.FSPath), nil

case TypeInMemory:
return NewInMemoryStore(), nil

Expand Down
76 changes: 76 additions & 0 deletions pkg/kv/fs.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
package kv

import (
"context"
"os"
"path"

"go.uber.org/zap"
"golang.org/x/sync/errgroup"
)

type FSStore struct {
fsPath string
}

func NewFSStore(logger *zap.Logger, fsPath string) *FSStore {
return &FSStore{
fsPath: fsPath,
}
}

func (s *FSStore) Start(ctx context.Context, g *errgroup.Group) error {
return nil
}

func (s *FSStore) Get(ctx context.Context, ns Namespace, key string) (string, error) {
path := path.Join(s.fsPath, string(ns), key)

b, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
err := s.Touch(path)
if err != nil {
return "", err
}
return "", nil
}
return "", err
}
return string(b), nil
}

func (s *FSStore) Set(ctx context.Context, ns Namespace, key string, value string) error {
path := path.Join(s.fsPath, string(ns), key)

err := s.Touch(path)
if err != nil {
return err
}

err = os.WriteFile(path, []byte(value), 0644)
if err != nil {
return err
}

return nil
}

func (s *FSStore) Touch(fpath string) error {
baseDir := path.Dir(fpath)
if err := os.MkdirAll(baseDir, 0755); err != nil {
return err
}

file, err := os.OpenFile(fpath, os.O_RDONLY|os.O_CREATE, 0644)
if err != nil {
return err
}

err = file.Close()
if err != nil {
return err
}

return nil
}
50 changes: 50 additions & 0 deletions pkg/kv/fs_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
package kv
Kuixz marked this conversation as resolved.
Show resolved Hide resolved

import (
"context"
"testing"

. "github.com/smartystreets/goconvey/convey"

"go.uber.org/zap"
)

func TestSpec(t *testing.T) {
Convey("Given a fresh context", t, func() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
test_path := t.TempDir()
logger := zap.NewExample()
store := NewFSStore(logger, test_path)
ns := Namespace("namespace1")

Convey("Values can be set", func() {
key := "key1"
value := "value1"
err := store.Set(ctx, ns, key, value)
So(err, ShouldEqual, nil)
})
Convey("When reading a set value", func() {
key := "key2"
value := "value1"
store.Set(ctx, ns, key, value)
returned, err := store.Get(ctx, ns, key)
Convey("The application does not error", func() {
So(err, ShouldEqual, nil)
})
Convey("The value read is equal to the value set", func() {
So(returned, ShouldEqual, value)
})
})
Convey("When reading a value that was not set", func() {
key := "key3"
returned, err := store.Get(ctx, ns, key)
Convey("The application does not error", func() {
So(err, ShouldEqual, nil)
})
Convey("The value is empty", func() {
So(returned, ShouldEqual, "")
})
})
})
}
Loading