[FEATURE REQUEST]: Invoking from Databricks notebook

[guruvonline]

Hi,

Instead of creating my job with setjar or spark-submit, can i run my application directly from Notebook ?

[imback82]

@guruvonline Notebook experience for .NET for Apache Spark is not available on Databricks, and I suggest requesting this feature to Databricks directly so that they are aware of the customer demand.

There is a databricks-connect experience you can try. The UDF is not working yet, but we are currently working with Databricks engineers to enable this feature, so you should be able to use it to submit C# UDFs in upcoming releases.

[guruvonline]

@imback82 , i have spark batch job with UDF and trying to attach microsof.sparxx.jar to cluster and then run my below notebook code (attached to above cluster) as below. it start my app but looks like doesn't wait for it to complete and notebook shows as command completed.
From you response it means it is not supported but just wanted to get explicit confirmation on what im trying.

import org.apache.spark.deploy.dotnet._

DotnetRunner.main(Array("myapp.zip","myapp"))

[imback82]

I haven't tried this, but I think it may work out (you need to specify hdfs location for your zip). Let me know how it works out.