Pydantic v2

[dan-fritchman]

Came faster than expected: https://twitter.com/pydantic/status/1674820177005887508

[daquintero]

Worth mentioning they have a pydantic.v1 notation for an intermediate migration. They describe it here

[daquintero]

Also apparently there is an automatic migration tool https://github.com/pydantic/bump-pydantic#installation

[growly]

(FYI) Ran into this when I accidentally had pydantic 2.1.1

  File "/home/arya/src/Hdl21/hdl21/__init__.py", line 12, in <module>
    BaseModel.Config.arbitrary_types_allowed = True
    ^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py3.11/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 205, in __getattr__
    raise AttributeError(item)
AttributeError: Config

[dan-fritchman]

Yep, they're definitely not designing it to be API-for-API compatible.

[dan-fritchman]

WIP on PR #214.

Noting:

• The places we do the most involved Pydantic-stuff are probably Instantiable and even more on Scalar.
  • I don't know how to make them work in v2, yet.
  • Scalar particularly does inline conversions of constructor arguments that are very, very widely used, and would be fairly painful to de-feature.
  • I opened Migration: Root + Custom Data Type pydantic/pydantic#9044 on the topic.
• PR Pydantic v2 Compatibility #214 is attempting to support v1 vs v2, by detecting and reacting to them at runtime.
• Other differences:
  • The pydantic configs are dicts instead of classes. Seems fine.
  • Dataclass __post_init_post_parse__ is deprecated; we change to __post_init__. Seems fine, I think, including in the case of continuing to work with v1.
  • Lots more detail in here: https://docs.pydantic.dev/latest/migration

[dan-fritchman]

PR #214 tests have a few holes left.
Smallish (I think) stuff:

• The combination of Python < 3.9 and Pydantic v2 doesn't work. This is due to the Annotated type added to the standard library typing module, which Pydantic v2 uses very centrally. Maybe there is some __future__ incantation that would make it work, yet to be discovered here.
• I added Python 3.12  #215 to the same PR, and quickly rolled back (required) tests of it. Separate topic for that issue.

Prefixed is the sole remaining real thing.
Prefixed uses Pydantic's "custom data types" feature, to enable things like:

@dataclass 
class C: 
  p: h.Prefixed = 1.1 # <= this gets converted inline

# So do these:
C(p=1)
C(p=Decimal("3.1415")

The custom data types feature is pretty well overhauled in Pydantic v2. Definitely for the better, but different. The way to do this now is something like:

# Just add this `Annotated` thing
Prefixed = Annotated[Prefixed, BeforeValidator(to_prefixed)

Quite a bit nicer way to get the custom validation and checking.
Problem is, that Annotated... thing (it's a "special form", much like Union) does not play nice with isinstance. And we have isinstance(thing, Prefixed) all over. Particularly we have tons of values of type, like, Union[int, float, Prefixed], in which case something like isinstance(Prefixed) is, if not a necessity, certainly very helpful.

Couple options I can think of:

• Somehow get Annotated and isinstance to play nice? I don't see how, and the Python docs and Google and StackOverflow results don't seem to either.
• Replace all the isinstance call sites with some custom check for "Prefixed-ness". Similar to how we look for a __paramclass__ attribute to determine whether something is_paramclass. Not sure how many isinstance calls are in user-land and would break.
• ...Don't have the custom validation for Prefixed? I believe when I added it, we did not yet have Scalar, which has its own methods for pydantic v1/ v2 compatibility. (Which did in fact require removing any isinstance calls.)

[dan-fritchman]

Prior comments spawned some other, smaller issues including:

• Python 3.12  #215
• Python 3.7-3.8 and Pydantic v2 #217

And the resolution on Prefixed is "don't do custom validation for Prefixed".
And with that, PR #804 is just about there, hopefully finishing verification now.

[daquintero]

Hi @dan-fritchman, thanks so much for all the effort you have put into this, it's really exciting to explore the integrations this will enable. I was wondering, are #218 and #215 the blocking aspects for, say, a pre-release of a pydantic v2 version? Depending if 3.7 and 3.8 support is depreciated assuming they both are reaching end of life #217

[dan-fritchman]

Here is how I am thinking about what must (and might) be in the next release: #220

[dan-fritchman]

@daquintero where is that Python version/ release chart from BTW?
Looks pretty helpful.

[daquintero]

Ahh this is great for the next release, thanks!

The diagram is from here https://devguide.python.org/versions/ and it's really helps managing the target support for releases imo

[dan-fritchman]

Terrific that is quite helpful.
And I'm going to refer further commentary on the topic to #217.

[dan-fritchman]

Live in v6 at https://pypi.org/project/hdl21/6.0.0/