Best practices for integration of streaming datasource

[sapcode]

Dear Dagster community,

What is the best pipeline design for consuming a streaming datasource?
Lets say we have a data stream via SSE server side events where we get 1 MB per Minute.
We would like to store the stream continuously in S3 and write some logs every 1 Million records.
We found other discussions threads about signals and jobs picking up files but this was not recommended.

Could you provide an example in the documentation how to realize streaming datasource consumption?
Best regards

[alangenfeld]

The best tool at this time is using a Sensor https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors#sensors to consume the stream at regular intervals.

[sapcode]

Hi Alan,

can you please make an example code how to ingest a permanent data stream with a sensor?

We found this thread with IOManager code which looks promising:

#9243

Is that IOManager approach the way to go or the sensor approach for permanent streaming sources?

Best regards

[C0DK]

Is there any plans to fully support streaming in the future or will "micro batching" (I.e with sensors) the future preferred solution?

[sryza]

@C0DK - do you have thoughts on what it would look like for Dagster to fully support streaming?

[C0DK]

No probably not. it'll probably just be an asset that is always up-to-date. But micro-batching is probably just the way to go.

[geoHeil]

https://github.com/feldera/feldera https://www.youtube.com/watch?v=cn1Yaxwl6x8

managed via dagster might be super interesting