Proposal: persisting the autopostings to journal files would allow for review of changes made to rewriting rules

[lestephane]

Is your feature request related to a problem? Please describe.

I use autoposting rules to keep track of some out-of-pocket expenses that my company owes me.
If I make a change to an autoposting rule, I would like to review the ripple effect of the change, and of course version control it (not just the rule, all affected postings). If a change has many ripple effects, I'm much more likely to spot a problem at the gif diff stage. Currently I only see a problem when running hledger-ui after painfully looking for the transaction. If there is a problem with hledger itself (as when the synthetic transaction contains amounts that are formatted in an ambiguous way), I can't do anything, as the transaction does not 'exist' on the filesystem. This is how for example, a sum of 2.500 EUR from an autoposting gets interpreted as 2.5 EUR and I can't do anything about it, I can only specify a multiplier (*1, *-1, etc). If I could at least have these autopostings generated and visible in a file somewhere, i could apply an additional regex based search and replace in my build pipeline.

Version and Runtime Information

Please mention the version number of hledger-flow you are using:

$ hledger-flow --version
hledger-flow 0.12.3.0 linux x86_64 ghc 8.6

Is this the latest version? YES

If your request includes commands you ran and the output, please also include
the runtime options with --show-options e.g:

Describe the solution you'd like

• one new file that is looked for by hledger-flow (besides construct and preprocess) -> rewrite-rules (name inspired from https://hledger.org/manual.html#re-write-rules-in-a-file's rewrite-rules.journal file parameter example)
• hledger-flow includes import works in all respect as before, except...
• ... if rewrite-rules is found, the journal generation command for 3-journal includes
  • the --auto command-line option
  • an additional -f command-line option pointing to the rewrite-rules
• rewrite-rules is not included in YYYY-include.journal, since the 3-journal files then already contain the side-effects of applying the rewriting rules.

(Note: I don't know how the 3-journal files are generated today, and I can't see any 3-journal related output in the verbose logs, so I indicated how I would do it using command-line options to the hledger print command)

Describe alternatives you've considered

The way I would do this today is by invoking

$ hledger-flow import
$ hledger print --auto > before.txt
(make a change to autoposting rules)
$ hledger print --auto > after.txt
diff before.txt after.txt

If the aggregated journal is rather large, it will become impractical to run a diff on it.

Potential problems
If rewrite rules are put in some other shared include file (à la #50) and not rewrite-rules files, hledger-flow might not detect that --auto needs to be added when generating the journals.

The behaviour is undefined in that case. The user is expected to put autopostings in a file by that name, which should standardize things. Since hledger-flow is opinionated anyway, that should be fine.

[apauley]

Interesting use case - I haven't yet used auto postings. I'd like to add some to my personal journals so that I can run into the same issues you're describing.

The next week or two will be difficult though - but you're welcome to remind me about this if you haven't seen any activity in a few weeks.

[lestephane]

I don't use autopostings much yet, but seeing the effects of my changes in journals would allow me to experiment much more with them.

[lestephane]

This is not urgent

[lestephane]

I have the following (working) workaround

First, make it possible to override the hledger command in the scope of my root import directory

(in the root import directory)
$ cat .envrc 
PATH_add $(expand_path .)

Then, override the hledger command in such a way that hledger-flow picks it up

$ cat hledger
#!/usr/bin/env bash

MYDIR=$(readlink -e $(dirname "$0"))

set -euxo pipefail

case $@ in
    print\ *--file\ -*)
        cat "${MYDIR}/explicit-autopostings.journal" - | "${HOME}/.local/bin/hledger" "$@" --auto
    ;;
    --version|--help)
        "${HOME}/.local/bin/hledger" "$@"
    ;;
    print\ --forecast*)
        "${HOME}/.local/bin/hledger" "$@"
    ;;
    print\ --rules-file*)
        "${HOME}/.local/bin/hledger" "$@"
    ;;
    *)
        echo "unhandled: " "$@" 1>&2
        exit 1
    ;;
esac

A few notes:

• The rewrite rules are in the shared explicit-autopostings.journal file, which itself is not
  !include-d anywhere else
• Adding an extra -f explicit-autopostings.journal command line argument did not work.
  Instead, I add to pipe the autopostings to the hledger command in addition to the original input,
  hence the cat "${MYDIR}/explicit-autopostings.journal" -
• There is an additional --auto argument added to generate the automated postings.

As a result, my 3-journal target directories now contain the automated postings, and I can see the effect of my changes to rewrite rules in all their glory / infamy. I've got regressions on almost all the automated postings, so it was well worth adding.