4.x Release plans

[andrewgsavage]

Most of the topics in #201 are completed so I'm starting this for the next release, which I presume will be 4.0.0

Skimming through the previous topic and development plans, it looks like the following weren't resolved and could be in the next release:

• Linting, precommit
• Renaming/aliasing of UFloat, AffineScalarFunc

I've been working on:

• Some reorganisation into smaller files Reorganise core #187
• improved numpy support (https://github.com/andrewgsavage/uncertainties/tree/numpy will submit a PR after Reorganise core #187 is merged)

Is there anything else you'd like to see in a 4.0.0 release?

[newville]

@andrewgsavage This all sounds great to me. I would still like to have something done about the self-referential
LinearCombination thing of:

class Variable(AffineScalarFunc):
     __slots__ = ('_std_dev', 'tag')
    def __init__(self, value, std_dev, tag=None):
        value = float(value)
        super(Variable, self).__init__(value, LinearCombination({self: 1.}))

with the lengthy discussion at #218 (somewhat noisy, as we are still coming to grips with what this means).

I think the simplest solution might be to replace LinearCombination({self: 1.}) with LinearCombination({None: 1.}) and then have AffineScalerFunc pass in its own self to LinearCombination.expand()as the "value to use in place ofself". I think that will make Variablebe easier to serialize, and not need the wholeid` business. I suppose that means I just volunteered to make a PR...

[wshanks]

I would be opposed to making std_dev un-settable (n and nominal_value are).

For what reason do you oppose making std_dev un-settable like nominal_value? Is it to preserve the feature of being able to modify an independent variable's uncertainty and calculate the new uncertainty for a dependent AffineScalarFunc? Or to keep backwards compatibility?

I would also be okay with #219 or something like it if we could add a warning to std_dev.setter of Variable that, if the user tried to change the value to or from 0, would complain that previous tests of equality and hashing might no longer be valid. That was the main issue for me -- a hash should never change and items that hash to the same value need to be equal, but changing a finite std_dev to 0 would make an AffineScalarFunc equal to a float with the same nominal value. Tweaking the value in any other way should be fine and with that warning the user could even set the value to 0 knowing they should try to use it as a dict key afterwards.

[newville]

@wshanks Very sorry, my mistake. I would NOT be opposed to making std_dev an un-settable property, just as n and nominal_value are unsettable. I do worry a bit that making _std_dev and _nomimal_value static might make implementation more complicated -- perhaps making these __std_dev__ etc would make them more obviously "private".

[jagerber48]

@newville @wshanks See #251

[andrewgsavage]

I have a hunch that core is architecture the way it is to support a late-evaluation of chains of dependencies between variables with uncertainty in a way that allegedly improves performance. I haven't seen any performance data anywhere, it's just a hunch I have based on the comments.

I think it would be worth seeing either trying an instant evaluation or using a covariance matrix and seeing how much slower it would be to understand the performance cost of a simpler implementation

[jagerber48]

@andrewgsavage after re-implementing the "linear combination of uncertainty atoms" and finally understanding how it works, I less have the feeling that it is unintuitive compared to a covariance matrix approach. And I think it's a bit nicer programmatically because there isn't shared information between different objects. In the linear-combination-of-uncertainty-atoms approach each variable keeps track of the atoms it depends on. Then when two variables come in contact their local atoms are compared.

In the covariance matrix approach there is a sort of global state that needs to be kept track of which variables exist and how are they correlated. There could be one huge covariance matrix that keeps track of all variables, but I think this would quickly become untenable if large calculations are done. One could imagine only storing the covariance between variables X and Y locally on those objects, but this would need to be synchronized, and, depending on the connectivity of calculations, we could also end up with huge covariance matrices.

In the linear combination of atomic uncertainties we write X = X_0 + dX, Y = Y_0 + dY Z = X + Y = (X_0 + Y_0) + (dX + dY). In the covariance approach we think of only Z = X+Y` but retain memory of the covariances between X, Y, and Z.

I have ideas how this can be more clearly documented.

---

However, you also mentioned testing early vs. late evaluation. I do think early evaluation could simplify the implementation, so I agree that this would be interesting.

[newville]

@andrewgsavage @wshanks @jagerber48 But also: is it OK if we view #245 as something we should merge now and then push out version 3.2.2 for Numpy2 compatibility, and then switch to more refactoring for 4.0?

[wshanks]

That sounds good to me!

[andrewgsavage]

Yes please, numpy2 support warrants a release as soon as it's ready

[wshanks]

One thing I think would be good to tackle for a 4.x release is the numpy.matrix deprecation warning (#114). I think the reason numpy wants to deprecate numpy.matrix is that the main motivation for it was so that * could be overloaded as matrix multiplication, but after pep425 added the @ symbol for matrix multiplication that motivation went away and it has just been a very slow deprecation in numpy as it waits for scipy, etc. to stop using numpy.matrix (it passed on the opportunity to complete the deprecation with numpy 2.0).

The problem for Uncertainties is that we have umatrix which uses numpy.matrix. We should figure out if we want to keep or deprecate umatrix. We could switch to keep the current umatrix interface without using numpy.matrix or we could deprecate it and just keep the inverse function and let users call that on a plain uarray when necessary. I think the inverse function is the main value added by umatrix. We don't need to make many changes for 4.0, but I think it would be good to add a deprecation warning on umatrix with that release if that is the direction we want to go.

[newville]

@wshanks I agree. Numpy encourages abandoning matrix. We should listen to them. I am +1 on deprecating umatrix either now or with 4.0, and hopefully removing it in a year or so.

[jagerber48]

I am +1 on getting rid of umatrix. Right now #262 doesn't fully support umatrix. Right now #262 mostly acts as if np.matrix doesn't exist. If necessary, I can modify it so that it checks if inputs are matrices and, if so, it will recast them to matrices during derivative calculation and at the output. At least supporting inv and pinv in this way should be not too hard to do.

I would love if we could just drop umatrix at 4.0 but maybe that is unfair to users and we must deprecate it for some time before entirely dropping that? Basically the question is should I put in the work to make sure #262 supports umatrix at the level it is currently supported?

[jagerber48]

I want to ask here about timing for switching to sciform as a formatting backend. My suspicion is that, if this change is approved, it will likely come with at least some changed formatting behaviors, and possibly some breaking changes to the format specification mini language. If that it is the case I would say it is likely we would want sciform to be added at a major version change.

However, I fear this change will not be ready for 4.0 just given timings of everything. When/if #262 is completed I think it won't be long before we can knock out some smaller tasks to be ready to release 4.0. The question is, would we want to wait for the possible sciform code changes and discussions to go through for 4.0 or not? If not, is there a chance for sciform to come in in a later version? 4.x or 5.0? What I'm wondering is if 4.0 is the last ship that sciform can sail in on and I should try hard to get it in, or if I can wait and do sciform on whatever timeline is "comfortable" for myself and others.

Also apology to folks for the huge flurry of activity from myself on JUST TODAY. this happened to be a day that a lot of my work coalesced and I was able to spend some time on github. Please respond at your leisure.