sparkmon is a Python package to monitor Spark applications. You can see it as an advanced Spark UI, that keeps track all of Spark REST API metrics over time, which makes it quite unique compared to other solutions (see comparison below). It is specifically useful to do memory profiling, including Python UDF memory.
Monitoring plot example:
Disclaimer: Be aware that if you run Spark in local mode some of the subplots will be empty, sparkmon is designed to analyse Spark applications running in a cluster.
- Log the executors metrics
- Plot monitoring, display in a notebook, or export to a file
- Can monitor remote Spark application
- Can run directly in your PySpark application, or run in a notebook, or via the command-line interface
- Log to mlflow
This package brings much more information than Spark UI or other packages. Here is a quick comparison:
- sparkmonitor:
- Nice integration in notebook
- Doesn't bring more information that Spark UI, specially not memory usage over time.
- sparklint:
- Need to launch a server locally, might be difficult on-premise. sparkmon doesn't need to have a port accessible.
- Monitors only CPU over time, sparkmon monitors everything including Java and Python memory overtime.
- No update since 2018
- Data Mechanics Delight:
- Really nice and complete
- But cannot work fully on-premise
- Is not fully open-source
- Sparklens:
- But cannot work fully on-premise
- Is not fully open-source
- Python
- Spark
- mlflow (optional)
You can install sparkmon via pip from PyPI:
$ pip install sparkmon
$ pip install sparkmon[mlflow]Simple use-case:
import sparkmon
# Create and start the monitoring process via a Spark session
mon = sparkmon.SparkMon(spark, period=5, callbacks=[
sparkmon.callbacks.plot_to_image,
sparkmon.callbacks.log_to_mlflow,
])
mon.start()
# Stop monitoring
mon.stop()More advanced use-case:
import sparkmon
# Create an app connection
# via a Spark session
application = sparkmon.create_application_from_spark(spark)
# or via a remote Spark web UI link
application = sparkmon.create_application_from_link(index=0, web_url='http://localhost:4040')
# Create and start the monitoring process
mon = sparkmon.SparkMon(application, period=5, callbacks=[
sparkmon.callbacks.plot_to_image,
sparkmon.callbacks.log_to_mlflow,
])
mon.start()
# Stop monitoring
mon.stop()You can also use it from a notebook: Notebook Example
There is also a command-line interface, see Command-line Reference for details.
SparkMon is running in the background a Python thread that is querying Spark web UI API and logging all the executors information over time.
The callbacks list parameters allows you to define what do after each update, like exporting executors historical info to a csv, or plotting to a file, or to your notebook.
Contributions are very welcome. To learn more, see the Contributor Guide.
Distributed under the terms of the MIT license, sparkmon is free and open source software.
If you encounter any problems, please file an issue along with a detailed description.
This project was generated from @cjolowicz's Hypermodern Python Cookiecutter template.