WIP: Pygfx histogram

[gselzer]

This PR throws together a WIP PyGFX histogram widget using fastplotlib. The goal would be parity with the VisPy widget, but the current progress is quite far from that goal. Eyes and discussion encouraged.

TODO:

• Implement remaining HistogramCanvas methods
• Depend on a reliable version of fastplotlib - for this to work, you must currently pip install git+https://github.com/fastplotlib/fastplotlib.git@f0f80c6a4deabb0941154584c854f2004b0cc316. I could not find an earlier version of fastplotlib with axes.
• Support cmap changes
• Debug clims - for example, the mesh in between the clims does not show up until you first move an edge.
• Improve axes - it's hard to read both of them.
• Cleanup
• Test

Points of discussion:

• Is the fastplotlib dependency okay here? I know that a selling point of this application is "minimal dependencies", and fastplotlib brings in imgui-bundle, which is ~30MB. Granted, it's an optional dependency (i.e. you could use vispy instead).
• Gamma is currently "unavailable" for pygfx images, so we don't have it on the histogram (@tlambert03 maybe we need an issue for this?). But, if we did add gamma support for images, we'd probably want to subclass the LinearRegionSelector to add in a gamma curve.
• Is the line currently being used okay? It's so dense for me that (as seen in the gif) it almost looks like a mesh. I do think a mesh would be preferred, but fastplotlib doesn't really support these yet.

[tlambert03]

Woohoo!

[tlambert03]

thanks @gselzer, couple thoughts after pulling and running:

• it's very nice to see a working example :)
• indeed, it is quite far from parity with the vispy interface. I'm definitely open to some differences, but in general i'd say I prefer what we've done with the vispy interface to the one here. I prefer the clim handles and mouse interaction behavior there.
• it definitely is a bummer that this would have to bring in imgui. Why is that? At first i didn't install from git main, and it worked kinda ok, though the axes were different of course. what has changed between the last release and the current main that now requires imgui? From the source, it looks like they're using it for button elements, like reset/min max... is that true? It does start to become more like an app-backend (and bring in more dependencies) and less like a canvas/graphics backend that way.

Is the line currently being used okay? It's so dense for me that (as seen in the gif) it almost looks like a mesh. I do think a mesh would be preferred, but fastplotlib doesn't really support these yet.

• yeah ... barplots and mesh would be better here. no

I guess, all in all, it's nice to see something working, but the additional imgui dependency, very different interface, and lack of flexibility with bar plots do make me wonder what it would take to go just a little lower and use pygfx directly (I know that makes axes harder ... )

[tlambert03]

however ... it is better than nothing, and as long as we can keep it working with pygfx without the dependency on fastplotlib and imgui, and gracefully fallback (i.e. not showing a histogram button if the additional stuff isn't there), then it's fine with me as an experimental feature

[kushalkolar]

Take a look at this, and there are a few examples in the docs too which avoids having to make a subclass to do this: pygfx/pygfx#778

[kushalkolar]

the parent graphic here would be the line.

[kushalkolar]

Cool to see you guys doing more on this!

Depend on a reliable version of fastplotlib - for this to work, you must currently pip install git+https://github.com/fastplotlib/fastplotlib.git@f0f80c6a4deabb0941154584c854f2004b0cc316. I could not find an earlier version of fastplotlib with axes.

Yes we do not have a release with axes yet. We're in alpha (the entire WGPU ecosystem is still new and evolving and we're built on that). Axes are especially still quite wonky, we're really waiting for pygfx/pygfx#495 before they can work nicely (for an example of weirdness see fastplotlib/fastplotlib#613). Nonetheless it's nice seeing you try things to get more examples of use cases! We're planning for a new release in early January. We're catching up with upstream changes from pygfx and wgpu-py, and all the previous wgpu GUI code is now in rendercanvas.

Debug clims - for example, the mesh in between the clims does not show up until you first move an edge.

It might be:

• the z position
• things can get weird if the range of the space is very small or very large, I think this is why we scaled the histogram edges between 0-100 in our implementation.

Improve axes - it's hard to read both of them.

If you change the zoom level of the controller that should be sufficient.

fastplotlib brings in imgui-bundle

imgui is intended to be completely optional, if you are building a Qt app you probably don't need imgui. fastplotlib@main currently imports imgui but this PR will fix that.

Gamma is currently "unavailable" for pygfx images

I think it should be easy to add this to the shader? Not sure the best place to add it but if you post an issue they can probably help you.

Is the line currently being used okay? It's so dense for me that (as seen in the gif) it almost looks like a mesh.

The issue with thinner lines is that you have to be more accurate with your mouse movements to grab onto them, you could make a thinner inner line and a transparent thicker outer line and use the weighted_plus rendering mode.

[gselzer]

@tlambert03 made some progress here on a pure-pygfx version:

I'm still seeing a nice framerate, ~45FPS on my machine. Much better than the ~10FPS I was seeing with vispy.

Currently, I've got the gamma curve, but without a handle. The main reason I left it in is because it helps for clim identification - i.e. it's harder to see the clims without the gamma curve. It likely also provides some utility without the ability to edit.

If you like this direction better, there are some further bugs to squash:

• y-axis ticks are really tricky - the current pygfx Ruler implementation seems to struggle with short canvases, and I found that sometimes it wouldn't add any ticks - other times, there'd only be one tick. For now, I just add the maximum value as a tick, because it's easy and provides some information. I did play with manual ticks with the following lines, but I wasn't terribly happy with it - it would need further refinement to avoid tick overlapping:

  # e.g. for maximum_vin_value=456, produces [100, 200, 300, 400]
  step = 10 ** math.floor(math.log10(maximum_bin_value))
  self._y.ticks = [i for i in range(step, maximum_bin_value, step)]

• Currently pan/zoom is disabled. It's easy to implement (especially with @kushalkolar's suggestions!) but I'm thinking YAGNI more and more.
• Unfortunately, though, the mesh behaves strangely at such a small scale (not sure if this is what @kushalkolar mentioned about "range of the space ... very large"), where it looks like the values are changing when you pan/zoom, as you can (kinda, hindered by the low framerate) see in the GIF below:

[tlambert03]

great work as always! i like the general look and feel a lot better!

Currently, I've got the gamma curve, but without a handle.

by the way, even if pygfx doesn't have shader-side gamma, there's no reason we can't apply it on the CPU. vispy also didn't have shader side gamma until i added it in vispy/vispy#1844 ... and we also weren't using it here until #77 ... so CPU-side gamma is just fine as a fallback (that would be a different PR though)

I'm still seeing a nice framerate, ~45FPS on my machine. Much better than the ~10FPS I was seeing with vispy.

that's great! I still want to understand why thought 😂 ... particularly given that I didn't see it on my windows machine. also doesn't need to happen here, but some profiling to narrow it down to a specific culprit would be super interesting. (i.e. is it really actually vulkan vs opengl on your gpu? or something more mundane)

For now, I just add the maximum value as a tick, because it's easy and provides some information

I actually think that's great, and could very much live with just that for a while

pan/zoom is disabled. It's easy to implement (especially with @kushalkolar's suggestions!) but I'm thinking YAGNI more and more.

do you mean you think it's YAGNI to pan/zoom on the histogram? that's surprising to me... you mean you think it's fine to look at it full scale all the time? or just that it can be controlled programmatically rather than with mouse wheel? Also not sure how to reconcile that comment with the next video that seems to show pan/zoom? I think i'm not following :)

it looks like the values are changing when you pan/zoom

do they settle into the actual values after a very short delay though? if so, that also doesn't bother me too much

[gselzer]

by the way, even if pygfx doesn't have shader-side gamma, there's no reason we can't apply it on the CPU. vispy also didn't have shader side gamma until i added it in vispy/vispy#1844 ... and we also weren't using it here until #77 ... so CPU-side gamma is just fine as a fallback (that would be a different PR though)

Well, it does have a shader-side gamma, but it applies to the entire scene being rendered, which is unfortunate. But, more importantly, it's probably a separate issue that really pertains to the ArrayCanvas implementation (see #78). You're right that we could do it CPU-side as a stopgap.

that's great! I still want to understand why thought 😂 ... particularly given that I didn't see it on my windows machine. also doesn't need to happen here, but some profiling to narrow it down to a specific culprit would be super interesting. (i.e. is it really actually vulkan vs opengl on your gpu? or something more mundane)

I agree that figuring out why would be interesting, but until it affects someone else I feel like I could spend my time better elsewhere...

I actually think that's great, and could very much live with just that for a while

Great!

do you mean you think it's YAGNI to pan/zoom on the histogram? that's surprising to me... you mean you think it's fine to look at it full scale all the time? or just that it can be controlled programmatically rather than with mouse wheel? Also not sure how to reconcile that comment with the next video that seems to show pan/zoom? I think i'm not following :)

Well, I didn't push that code with the pan/zoom - I just added it back in to create the GIF 😅

I will push it shortly, though, because it does work.

I do still have a sense of YAGNI, in that full scale all the time might be okay. My concerns are more with how pan/zoom gets reset. Right now, changing the data always resets the view, and there is no dedicated UI control for resetting the view on the entire dataset. We may want a dedicated button for that. But I don't feel too strongly about it all.

do [the values] settle into the actual values after a very short delay though? if so, that also doesn't bother me too much

No, the values never settle. Consider the fact that the maximum value (shown by the y-axis tick) isn't really shown by the histogram mesh. Depending on how far you pan, you might see some bins close to that value but I never actually see one at that value. Would be interested to see if others can replicate this behavior.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 264 lines in your changes missing coverage. Please review.

Please upload report for BASE (v2-mvc@665135b). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/ndv/views/_pygfx/_histogram.py	0.00%	262 Missing ⚠️
src/ndv/views/_app.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##             v2-mvc      #76   +/-   ##
=========================================
  Coverage          ?   67.07%           
=========================================
  Files             ?       42           
  Lines             ?     4367           
  Branches          ?        0           
=========================================
  Hits              ?     2929           
  Misses            ?     1438           
  Partials          ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tlambert03]

I do still have a sense of YAGNI, in that full scale all the time might be okay

I zoom all the time with 16 bit data (in micromanager and elements). With full scale you normally don't have the sensitivity you need when adjusting the clim handles for data with low intensity values. So I don't see this one as yagni