Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EAGLE-1231: Added 'encoding' attribute to fields in Logical Graph schema #265

Closed
wants to merge 4,819 commits into from

Conversation

james-strauss-uwa
Copy link
Collaborator

@james-strauss-uwa james-strauss-uwa commented Jun 13, 2024

Added a new encoding attribute to each field in the Logical Graph schema. Attribute is supported by EAGLE.

Allowed values are:

  • binary
  • pickle
  • dill
  • npy
  • base64
  • utf-8

Default value is "dill"

awicenec and others added 30 commits May 2, 2023 16:54
Very small change, but seems to work now again
Added parser fields to doxygen of apps
The method has been split apart into 4 individual methods and the rest has then been integrated with the calling method ‘make_single_drop’. Much more clean now, but required a number of changes throughout the code to make all the tests and the graphs working again. 

Updated doxygen strings to get the new dropclass parameter for every builtin drop. ‘dropclass’ replaces all the individual *class (e.g. dataclass, appclass) parameters and made quite a bit of the code more straight forward. This had effects in many places around the code.

Updated categoryType and category to use both of them consistently throughout the code.

Changed to “name” everywhere. There had been a few places where this was not yet done.

Cleaned-up the construction of drop_spec, tried to use a dataclass, but that would have caused too many changes for little benefit.

adjusted all test graphs

adjusted all tests
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have skipped reviewing this pull request. We don't currently review these file types ['.schema'] - Let us know if you'd like us to change this.

@james-strauss-uwa james-strauss-uwa self-assigned this Jun 13, 2024
@coveralls
Copy link

Coverage Status

coverage: 79.484% (+0.02%) from 79.469%
when pulling 4ccb13d on eagle-1231
into db91b75 on master.

@coveralls
Copy link

Coverage Status

coverage: 79.469%. remained the same
when pulling aa73a5a on eagle-1231
into db91b75 on master.

"encoding": {
"type": "string",
"enum": ["binary", "pickle", "dill", "npy", "base64", "utf-8"],
"default": "utf-8"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming this default is a decision that's been made previously, as opposed to being arbitrary?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree in principle not to use utf-8. We did not have the ability to specify this per field, but only globally. The previous global default value was pickle, but I think we should switch to dill instead, since that can serialize a lot more than pickle. Also note that this only applies (like before) if a field is used as a port.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I'll remove "utf-8" from the options, and set the default to "dill".

@coveralls
Copy link

coveralls commented Sep 9, 2024

Coverage Status

coverage: 79.694%. remained the same
when pulling 84216fe on eagle-1231
into ea6cc06 on master.

@awicenec
Copy link
Contributor

Added a new encoding attribute to each field in the Logical Graph schema. Attribute is supported by EAGLE.

Allowed values are:

  • binary
  • pickle
  • dill
  • npy
  • base64

Default value is "dill"

Sorry, there obviously is a misunderstanding, what I meant is not to use utf-8 as the default, not to not have it at all. Could you please add it to the list again, since that is the only way to pass plain strings correctly.

@james-strauss-uwa
Copy link
Collaborator Author

@awicenec I've added 'utf-8' back to the options. And 'dill' is the default in the schema.

I'd be happy to merge this PR. I'm just a little concerned that github thinks it includes 4819 commits!

Would this be sorted out by the merge?

@myxie
Copy link
Collaborator

myxie commented Sep 12, 2024

@james-strauss-uwa If you modify the merge branch from master to something else, and then back to master, that may resolve the erroneous number of commits.

@james-strauss-uwa james-strauss-uwa changed the base branch from master to yan-973 September 12, 2024 03:31
@james-strauss-uwa james-strauss-uwa changed the base branch from yan-973 to master September 12, 2024 03:31
@james-strauss-uwa
Copy link
Collaborator Author

@myxie Thanks for the suggestion, but it didn't seem to work.

Do you think we should merge anyway, or (since it is such as small change) create a new branch and PR?

@myxie
Copy link
Collaborator

myxie commented Sep 12, 2024

@james-strauss-uwa on reflection the issue is due to the --force push that has been applied to the GitHub repo by @awicenec. This has likely over-ridden a significant number of commits that you have made and exist within the history of the commits within your branch. Ideally, we avoid --force push, as it leads to the complications discussed in https://www.atlassian.com/git/tutorials/merging-vs-rebasing:

This overwrites the remote main branch to match the rebased one from your repository and makes things very confusing for the rest of your team. .... Again, it’s important that nobody is working off of the commits from the original version of the feature branch.

The issue is that the force push has been applied whilst other branches were using the pre-existing commit IDs.

@james-strauss-uwa
Copy link
Collaborator Author

Created a new branch eagle-1231-2 and will create a new PR for that branch

@james-strauss-uwa james-strauss-uwa deleted the eagle-1231 branch September 13, 2024 05:11
@myxie myxie mentioned this pull request Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants