Schema Registry Support

[juanpedromoreno]

Currently, frees-rpc supports Avro serialization using avro4s. Ideally, it might be great if we support Avro serialization and, also schema versioning, allowing the evolution of schemas. One option would be addressing it by using the Schema Registry.

[L-Lavigne]

For reference, here are the corresponding issues for avrohugger which is our implementation for Avro-schema sourcegen:

Schema downloads: julianpeeters/sbt-avrohugger#26
Schema uploads: julianpeeters/sbt-avrohugger#27 (most of the discussion and code samples are there)

There might be something there we can leverage.

[L-Lavigne]

This sbt plugin could also be useful: https://github.com/iadvize/sbt-avro

[L-Lavigne]

Quick note - regardless of API used, the Schema Registry only supports .avsc files which are individual class definitions (our @message classes).

Some of our users have .avdl or .avpr protocol files which contain multiple class definitions, as well as RPC services (our @rpc methods) which aren't definable in .avsc format and thus not storable in the registry. So in order to support those formats we'd need to break them apart into .avsc's (there are tools for this) before uploading them, minus their RPC's, to the Registry. (Same goes for validation against the registry.) If we support downloading in addition to validation, we'd then need to re-assemble them into .avdl's or .avpr's to which we would then add locally-defined RPC call definitions.

[pirita]

most of the work can be done with:

• sbt-avro: Downloading/Uploading to schema-registry and conversions to case classes.

• avro tools: It allow us to make conversions between .avdl (not the RPC methods) and .avsc.

We just need to separate the model definition and the RPC methods. I am to sure if it is possible, but If we don't want to add RPC local-definitions, we can make a module to convert an .avsc with RPC definitions to an .avsc without RPC definitions.

For example, we can convert:

  com.company.package.Apple getApple(com.company.package.Request request);

to

  record getAppleRPCMethod {
    string com_company_package_Apple;
    string name;
  }

  record getAppleRPCMethodParameters {
    string com_company_package_Request__ParamType1;
    string request__ParamName1;
  }

[L-Lavigne]

Creating special records to represent RPC calls is an interesting idea! We could do this with a single record I think, and using the actual message record types as req/resp types instead of strings.

[pirita]

It could work.

  record getAppleRPCMethod {
    com_company_package_Apple getApple;
    com.company.package.Request request;
  }

I think is worth to try if it is possible to have a way to convert from a .avdl (models and RPC definitions) to .avsc and and vice versa.

[L-Lavigne]

Schema registry support (for Avro and more) will be implemented in the Compendium project: https://github.com/higherkindness/compendium.