Are the Spark.NET 3.5.1 works in Polyglot notebook for UDF examples?

[GeorgeS2019]

@grazy27
4. User-Defined Functions

Previously in older version, it is not possible to work in the UDF notebook. Not sure if the latest 3.5.1 overcomes this limitation

https://github.com/Apress/introducing-.net-for-apache-spark

[grazy27]

@GeorgeS2019 It does not.

It tries to get entire declaring type, which is anonymous assembly that only exists within interactive session context.
And fails with an exception, as result:

Code expects declaring assembly for UDF to be accessible:

Notebooks without UDFs and broadcasts work fine.

[GeorgeS2019]

Do U have recommendation to the polyglot team what changes they need to make to accommodate spark.net UDF to work?

I am going to invite them to address your findings

Good job

[grazy27]

Certainly, there are ways to achieve this.
In the context of Dotnet.Spark, we can determine whether the current code is running in an interactive environment and manually include library bytes in the command payload (or alternatively, save the library as a file and load it on the worker side).

PoC:

For this PoC to work, we also need to hardcode executing assembly as 1.dll

Polyglot team could do something similar on their side as well, but it'd have to be strongly names and uploaded to GAC, or explicitly target '%DOTNET_WORKER_DIR%'.
This would only work locally, and other dotnet.Interactive wrappers would need the same fix.

This issue falls primarily within Dotnet.Spark's scope of responsibility. This issue should be taken into account when addressing migration from BinarySerializer to somewhere else, since the responsible code will have to be rewritten.

[GeorgeS2019]

dotnet/interactive#3797

[wudanzy]

I guess pyspark did save all bytecode? We can do the same in spark.NET.

[grazy27]

I’ve made some progress in understanding the interaction, though it’s still not entirely clear. Here’s what I’ve gathered so far:

Dotnet.Spark contains a separate solution for this already, Microsoft.Spark.Extensions.DotNet.Interactive. It registers into .net interactive pipeline and adds all executables via 'spark.addFile' functionality. Does pretty much the same I did in my PoC, but also handles more special cases and uses native spark function for it.
That's clever, we don't have to serialize IL code and send it to spark.worker.
On a worker side, there's dedicated code for handling this as well.

The only problem i faced is that it doesn't work, it relies on Dotnet.Interactive API Utility.DisposableDirectory API, using which throws MethodAccessException.

According to this issue in dotnet interactive, is not ready for 1.0 stable release.

Thanks @GeorgeS2019 for creating issue in interactive, let's wait and see if they can point us with this exception.