ReleaseNotes.txt

=========== Changes for Bpipe Version 0.9.9.7 ============

   - Fix incorrect "abnormal termination" messages
     printed to console when pipeline stopped with 'bpipe stop'

   - Fix incorrect 'pre-existing' printed for outputs that were 
     created by pipeline

   - Fix genome not accessible in pipeline the first time downloaded, 
     printing error

   - Preliminary support for executing pipelines on Google Cloud Services
     (Compute Engine) and mounting storage for pipelines from Google Cloud 
     Storage

   - Preliminary support for executing pipelines on Amazon Web Services 
     using EC2 and mounting storage for pipelines from S3

   - The 'groovy' command can now run embedded groovy (executed outside
     Bpipe) using the groovy runtime bundled with Bpipe

   - Support aliasing to string values in addition to outputs 
    
   - Experimental support for beforeRun hook in command config: execute
     arbitrary groovy code before a command executes
    
   - Re-execute checks if a commmand in the same stage has executed 
    
   - synchronize initialization of dir watcher to fix sporadic 
     ConcurrentModificationExceptions 
    
   - Many performance improvements, esp. for large, highly
     parallel pipelines
     
   - Support configuration for number of retries for status
     polling of HPC jobs 
     
   - Fix empty embedded parallel stage list causing resolution of incorrect
     downstream input 
    
   - Add support for 'optional' to make input optional, and 'flag' to add flags 
     in commands eg: ${input.csv.optional.flag('--csv')}
     
   - Fix leak of 'var' variables across branches when 'using' applied to 
     pipeline stage 

   - Fix error if 4 or more arguments passed to "to" in transform 

   - Fix bpipe complaining spurious outputs not created on retry,
     but not original run 
     
   - Fix some bugs where branch names were not being observed 
     
   - Fix branch name sometimes inserted without separating period for transforms 
    
   - Avoid redundantly putting branch name into files 

   - Improved detail in error / log messages in a few places 
     
   - Add region.bedFlag(flag) method for convenience when passing 
     regions to commands 
    
   - New operator: merge point operator (>>>) automatically configures a stage
     to merge outputs from a previous parallel split

   - Fix missing branch and '..' in filenames 
     
   - Change: globally defined variables must now be held constant
     once pipeline starts 
     
   - Fix split regions not stable between runs, set region id as branch
     name 
     
   - Fix bed.split producing different splits if run
     repeatedly on same bed 
     
   - Fix errors output if SLF4J referenced in user loaded libraries
     
   - Fix npe / improve error message when filter used with mismatching output ext 
     
   - Fix error in stage body resulting in confusing 'no associated storage'
     assertion failure 

   - Support for 'var' expressions in main pipeline script to define optional
     variables 
     
   - Add 'allowForeign' option to 'from' to let it process non-outputs 
     
   - Lessen the retries and retry interval when file cannot be cleaned
     up 

   - Support for sending to Gitlab using 'send' command

   - JMS support now responds to 'ping' message with 'pong' reply
     if JMS 'Reply-To' is set to allow for status monitoring
     

=========== Changes for Bpipe Version 0.9.9.5 ============

  Features:

   - Support for Bpipe 'agent' which can listen to requests via
     JMS (activemq for now) to run pipelines
     
   - Support sending json to JMS/ActiveMQ via send command 
     
   - Support for sending JSON content to URLs via send command
     
   - Utilitity function 'asQuotedList' formats lists of inputs
     as syntax compatible with js,python,groovy scripts 
     
   - Experimental support for "sequential" or power operator 

   - Print out the outputs to be created when run in test mode, 
     in addition to the command to run (existing behavior)
     
   - Support for post-command hook to execute a script after every 
     command 
     
   - Add command start and stop time to output from 'bpipe query' 
     
   - Add start and stop time to display in 'bpipe errors' 
     
   - You can now specify java libraries as list in bpipe.config,
     in addition to the existing string form
     
   - Support for check review states, and improved check display 
     

  Fixes:


   - Fix exception/error if double input extension overlaps 
     output with sng ext, eg. $input.foo.bar  happens to match
     upstream $output.foo
     
   - Fix completed event sometimes not sent for failed stages 
     
   - Fix failed checks written to std out at end of run 
     
   - Fix situations where checks are re-executed when not 
     required 
     
   - Support supplying input files explicitly to run in 
     run([...])
     
   - Support for JMS notifications of pipeline events
     
   - Fix error displayed by 'bpipe log' if no pipeline has ever 
     been executed in dir 
     
   - Prevent recursive load caused defining a segment inside a 
     loaded file
   
   - Add segment structure to JSON pipeline representation
     Previously segments were expanded into their constituent stages 
     in the JSON representation, and thus the higher level pipeline
     structure was lost.  Now each node and edge in the group
     is given a type, and both the segment and its intenral
     structure are output as nodes / edges.  
     
   - Fix sometimes getting wrong last run pid (thinks still 
     running) 
     
   - Change to make error in loaded script terminate pipeline 
     instead of just printing error and continuing
     
   - Fix bpipe log stops working after some commands until bpipe run 
     again 
     
   - Avoid printing out cleanup message if zero files to cleanup
     
   - Move some of the docs out of Language and into Guides 

   - Prevent exception in bpipe errors if command state not saved 

   - Fix 'bpipe errors' not showing log output if hyphen in branch 
     
   - Fix status command, add regression test 
     
   - Groovy command in script now handles embedded quotes,
     which previously caused syntax errors
     
   - Fix error when nested segment made of single stage defined 
     

Changes for Bpipe Version 0.9.9.2 
===================================

This release has focused primarily on performance, especially
for pipelines involving high concurrency and large numbers
of inputs and outputs. To achieve this, Bpipe now depends
on new Java APIs available in Java 1.7. This means that
Bpipe now requires Java 1.7 and will not work any longer
with Java 1.6.

Additionally, a range of small bugs have been fixed and 
smaller features added. 

## Backwards Compatibility

Of note, there is one change that can
affect backwards compatibility. This relates to use of the 
'transform' construct when multiple files of the same type are
transformed to multiple files of a different type. In the past
you could write a single transform such as:

```
transform('.txt') to('.csv') {
    ...
}
```

Previously this would transform every text file in the inputs
to a corresponding CSV file as output. From 0.9.9.2 forward
this will only transform a single input file. For multiple files, the
transform must now be specified as:

```
transform('*.txt') to('.csv') {
    ...
}
```

This change has been made to resolve ambiguity, since there was no 
way previously to specify that a stage transforms exactly 1 input file
of a given type.

## Change list

   - Large scale refactoring for performance with
     large numbers of inputs / outputs.
     Bpipe now requires Java 1.7!

   - Issue #180: bpipe doesn't allow empty branch list 

   - Issue #184: chr not passed to nested parallel pipeline 

   - fix wrong output referenced for 2nd double ext in
     produce eg: $output.bar.txt 

   - support for modules in TorqueCommandExecutor 

   - fix potential failure to cleanup files on error 

   - Issue #175: Check state can be confused b/w branches 

   - Handle errors in log init so that log fail doesn't prevent
     ability to run Bpipe (java logging tries to obtain exclusive 
     locks which aren't supported on some file systems. 

   - Update to groovy 2.4.6 (helps performance)

   - Fix ambiguous behavior of transform() when multiple inputs
     match transform pattern. New behavior requires transform('*.ext') to
     match upstream multiple files ending with .ext (minor possibilty for 
     backwards incompatibility)

   - Fix output resolved as input for produce+double ext + numbered input 

   - Fix exception  when check saved without pass/fail set 

   - Fix leak of tail / sed processes under OSX in some
     situations

   - Fix bootstrap theme for docs not recognised 

   - Support to open a debugger Groovy console at any point 
     in a pipeline using debug() function

   - Add settings.gradle to stop build failing if
     checked out to directory other than 'bpipe' 

   - Fix possible hang when multi command used inside
     produce(...) 

   - support for "mem_param" for torque executor
     this allows to choose which parameter Bpipe should use
     between "mem", "pvmem", etc.  by setting mem_param="mem"
     in a command config 

   - Issue #176: support multiple commands per config in multi 

   - Various documentation fixes and cleanups

   - fix double file extension on input not resolving second and
     later input (eg: when referenced in form $input2.foo.bar)

Changes for Bpipe Version 0.9.9 
==============================

**NOTE:** see 0.9.9_beta_1 changes for further changes since previous release

   - SGE: Merge branch 'sge_fixes', including:

     * Fix #156, #142, Bpipe may hang or ignore job cancel / delete
       using SGE executor 

     * add -terse to sge default command options

     * change to support separate options:
       'sge_pe' and 'procs' rather than combined.  
       (Old combined form still supported for back compat.) 

     * jobs are created using user modifiable command template,
       specified via jobTemplate configuration parameter
       which can be object with string value of path to template,
       or a closure which will be executed dynamically to obtain the
       path

     * fix incorrect reporting that stop command
       failed in sge 

     * use Bpipe resolved working directory instead of $PWD since
       it is not guaranteed to be defined
        
     * fix cases of failure due to string procs variable 
       getting overridden with integer (dynamic calculation
       of $threads variable)

     * fix incorrectly sending slots as well as pe for some
       configurations 

     * support individual parameters for SGE options instead of compiling 
       them into 'additional_options' 
        
   - Fix choosing "terminate" not working for some executors 
     (issue #162) 
     
   - PBS-pro: combine multiple resource limits into single 
     statement, add support for procs, memory requests
     
   - Experimental option for some executors to disable buffering of 
     standard output, applies to torque, slurm, pbs-pro. Set "stdbuf" as 
     option to have bpipe run commands using 'stdbuf' command to disable 
     output buffering, when output from commands is delayed in appearing
     in Bpipe log.

   - Performance optimisations for large / complex pipelines:

     * Cache output graph in serialised form to avoid reading property 
       files at startup 
     
     * Reduce massive logging of outputs when large number of
       inputs and outputs in single stage 
    
   - Add 'bpipe errors' and 'bpipe log -e' command to 
     show output for failed commands 
     
   - Add support to display per command output using the 
     bpipe log -c <cmd id> 

   - Fix 'bpipe test' printing exceptions and hanging on 'multi'
     statement 
    
   - Fix regression causing possible failure when stage has 
     outputs but no inputs 
     
   - update gradle wrapper script, include changes omitted from
     2.0 upgrade 
     
   - Fix files in subdirectory not resolved when whole file name 
     matches input extension 
     
   - Fix variable specified with 'using' not overriding global value 
     
   - Fix 'check' results possibly leaking between branches 
     
   - Fix reading output.dir causing potential modification of
     output directory 

=========== Changes for Bpipe Version 0.9.9_beta_1 ============

   - Fix content type not set causing errors with some smtp
     connections 
    
   - Issue #155: Update to gradle 2.0 to avoid compile failure with JDK9 
    
   - Performance optimisations for large numbers of inputs /
     outputs 
    
   - Should report error if is referenced and
     finds no inputs 
    
   - Give a hint about which 'from' clause
     pattern did not find a match in error message 
    
   - Fix typo causing exception while logging error message 
    
   - Fix: setting branch name from within stage causes incorrect
     branches downstream 
    
   - Fix input resolved as output in rare circumstances 
    
   - Fix transform ...  to followed by double
     output variable extensions (.foo.bar) failing with confusing
     error message 
    
   - Fix procs not set in torque job when using variable 
    
   - Make bpipe return exit code reflecting whether pipeline 
     succeeded or failed 

   - Fix: bpipe may fail to cleanup files created in subdirectories 
    
   - Fix hang / exception when using multi command (regression) 
    
   - Fix regression caused by incorrect synchronization on wrong object.  Refactored
     Concurrency resource auctions for clarity 
    
   - Fix regression / failure when no inputs specify any fixed thread
     requirements 
    
   - Better / fairer scheduling for dynamically allocated $thread variables 
   
   - Support for ranges in procs parameter, eg: procs = 1..10, or
     uses(threads:1..10) { ...  } 

   - Fix: failure to parallelize stages inside a list added to 
     another list, eg: [foo, bar] + [fubar,fug] 
    
   - Added explicit license file for clarity of
     licensing 
    
   - Issue #148: bpipe parsing commandline memory
     specification incorrectly 

   - Issue #150: Cannot pass blank parameter via command line 
    
   - Various documentation fixes
    
   - Fix: junit tests were not executed 

   - Additional SGE options, SGE options moved inside script 

   - fix: commands run using torque / pbs not using -e flag for bash,
     thus only treated as failed if last command fails
    
   - support for aliasing inputs to outputs via 'forward <x> to
     <y>' to allow an input to be virtually treated as a named
     output within the pipeline 

   - fix: bpipe terminating if ssh session terminates while showing bpipe log 
     in foreground directly after initial launch 
    
   - save output from LSF (to assist in debugging failed LSF commands)
    
   - fix stages not found when defined in bpipe lib folder and referenced in
     segment declaration if used after load 
    
   - Switch to use html format for emails by default 

   - allow user to configure email format with 'format' attribute in
     configuration 

   - allow multiple templates per notification type with different formats 

   - fix problems in using 'send'
     and 'file' to send explicit files based on templates 
    
   - Fix high thread concurrency used when large number of concurrent branches 
     created by one pipeline branch 
    
   - Explicitly destroy process after waiting
     until streams closed to ensure file handles not leaked 
    
   - fix some cases where Bpipe could report file as missing when
     created in slow syncing distributed file system
    
   - Allow more screen width when using 'bpipe checks', better formatting 
     of check display 

   - Add option to list checks rather than enter interactive mode 
    
   - fix some cases where commands might not
     be stopped, and outputs not cleaned up when using bpipe
     stop.  Update saved command info after cleanup.  
    
   - ensure stop time is always set for commands 

   - close possible file handle leak in (undocumented) capture command

   - more robust checking for file existance 

   - refactor file existance checking into common utils method, to allow 
     wider application 
    
   - Avoid misleading termination message when user deliberately stops 
     pipeline 

   - Incorrect input used as basis for output file name in transforms inside 
     'from', if competing input available from previous stage
   
   - fix: rare deadlocks - use dedicated thread pool per pipeline tier to 
     avoid possibility of deadlocks in heavily nested pipelines

=========== Changes for Bpipe Version 0.9.8.7 ============

   - check /proc on linux rather than relying on kill 
     0 to see if pipeline is running (enables cross
     user support for "bpipe log") 
    
   - Addition of documentation to Bpipe distro
    
   - fix failure to cleanup files with no parent directory 
     (regression) 
    
   - Add __future__ import for Python 2 and 3 compatibility 
     (fix failure on Mac when Python3 is default)
    
   - switch to HTML format for default email.  to switch back
     to text, use notifications { smtp { format="text" } } in
     bpipe.config.  
    
   - Fix presence of subdirectory in bpipe output meta data folder 
     causing bpipe to spew errors 
    
   - Fix files sometimes not cleaned up on distributd file system 
     where lag exists in synchronizing b/w nodes

   - Issue #128: use qsub -N to name job according to stage/filename 
    
   - support for overriding a stage with an alternative implementation 
     within a particular branch 
    
   - fix: authentication failures on passwordless SMTP
     server 
    
   - fix failure in cleanup if output properties file
     did not get optional canonicalPath property added 
    
   - comment out verbose log message creating large bpipe
     logfiles for long runs 
    
   - avoid encoding whole directory path into internal property file 
     names (fix 'file path too long errors' on file systems allowing only
     256 chars in file names)
    
   - fix parent branches not visible in children via branch property 
    
   - fix method missing error when multiplying by map by list 
    
   - Reformatting and summarizing of errors at end of pipeline to make 
     it easier to see where & why pipelines failed 

   - Aggregating identical errors from
     multiple branches together in output so that failures from
     many branches are less verbose 

   - Support for 'load' within a pipeline stage.  Loads into the 
     context of current branch rather than global. Pipeline stages 
     not supported, only variables.
    
   - prevent error if files used by Bpipe are not writable but
     are deletable 
    
   - prevent premature wildcard expansion of
     quoted arguments passed to Bpipe    

   - support for cleanup as command in pipeline stage 
    
   - prevent error if files used by Bpipe are not writable but 
     are deletable 
    
   - attempt to ensure log messages written
     even if exit due to outofmemory error 
    
   - log message if file is not cleaned up because it doesn't
     exist: help to diagnose problems with files not cleaned up 
   
   - better log info about where config files are 
    
   - allow memory to be integer for torque/slurm/etc, fix error
     querying status in torque/slurm/etc, include queuing system
     command id in bpipe status output 
    
   - show pending and waiting jobs with bpipe status, don't throw ugly error 
     if no commands ever run 
    
   - Fix: downstream references to inputs
     by index throwing "expected more inputs" error incorrectly 
   
   - avoid confusing stack overflow when user defines branch
     variable with same name as pipeline stage 
    
   - accept 'MB' or 'GB' in command line option for memory,
     cache runtime directory instead of creating new File objects
     all the time to figure it out 
    
   - prevent null pointer exception when list containing null 
     provided to 'from' 
   
   - fix multiple inputs not
     forwarding correctly when referenced using "$inputs" 
    
   - Bug fixes for lsf, added walltime option for Lsf 
    
   - caching of DAG lookups by
     canonical path to speed up large, complex pipelines 
    
   - caching of canonical path with check of basePath to ensure
     consistency 
    
   - fix dependency issues after
     cleanup when non-default output directory used 
    
   - Performance optimisations for systems where canonicalPath is
     very slow, better logging to resolve dependency issues 

=========== Changes for Bpipe Version 0.9.8.6 ============

   - fix: errors may printed about files not existing when attempting 
     to move to trash
   
   - fix bpipe jobs not printing jobs after user ctrl-c's out of 
     run command

   - fix incorrect default for procs_mode variable for torque 

   - fix ugly stack trace error when transform called without arguments 

   - fix failing to escape trailing . in split patterns 
    
   - branch.dir now ha a default value so that it is easier to put in reusable
     pipeline stages 
    
   - Ensure walltime gets interpreted as
     integer even if passed as String / BigDecimal / Long etc 
    
   - fix trash option not working in bpipe cleanup 
    
   - Fix default output directory not applied to second
     produce argument if first specifies a directory 
    
   - fix produce() with reference only to output.dir causing
     retry test to incorrectly indicate commands would run again
    
   - Fix incorrect file reported for errors when multiple
     external files loaded, added some comments, removed hard
     tabs 
    
   - fix branch variable not overriding default value of
     variable specified with var 
    
   - Fix files not checked after 'produce' if previously output and then 
     removed outside of bpipe 
    
   - Fix files not checked after 'produce' if
     previously output and then removed outside of bpipe 
    
   - fix 'inputs' variable with double file extension not always resolving
     only files matching both extensions     

   - fix output() function causing spurious errors about incompatible inputs 
     when combined with output variable extensions 
    
   - fix command not running when explicitly specified by produce() if another
     implicit output referenced by command 
    
   - Support for produce using wildcard based on regex (via ~) 
    
   - Add information about total run time to report and command log 

   - support to specify branch filter on command line, to limit
     running to a specific branch (for testing, usually) 
    
   - fix explicit output() combined with produce(...) resulting in
     inconsistent behavior 
    
   - Fix stack traces printed out when errors occur to use actual name of user 
     scripts instead of groovy generated ones 

   - fix build.gradle not working with existing gradle wrapper 
    
   - Implement default behavior for transform with
     more to() values than from() values instead of failing 

   - Fix output referenced inside check causing check to be
     skipped / behave incorrectly 

=========== Changes for Bpipe Version 0.9.8.6_rc2 ============

   - Issue 100: report does not accurately reflect repeated and
     parallel stages 
    
   - fix: inputs passed in wrong order to parallel stage after 'produce' when 
     lexical sorting differs to input order 
    
   - enhance: migrated all of Bpipe to use groovy 2.3.4 for JDK1.8 compatibility
    
   - Refactoring of basis for pipeline diagrams to solve longstanding bugs 
    
   - fix: commands not appearing in html report 
    
   - fix: bad log statement causing error if log
     file fails to appear, better behavior for distributed
     file systems where command may finish before output file becomes
     visible on remote node

   - fix: transform() from() to() form can result in Bpipe not detecting 
     command failure 
    
   - fix: pipeline fails if pipeline stage is referenced both with and without
     'using' 
    
   - fix: internal errors (eg: outofmemeory) possibly leaving the Pipeline in 
     corrupted internal state 
   
   - enhance: Make it possible to set custom R executable via
     bpipe.config 
    
   - fix: some status codes from PBS/torque occasionally causing errors in 
     logs 
    
   - fix: local commands don't show pid with bpipe status 
    
   - fix: status command showing old jobs and /
     or failing in situations where command objects are missing /
     corrupt 
    
   - enhance: add thread id to log file (debug)
    
   - fix: not all outputs from branches split by file pattern forwarded to
     downstream stage when number of files differs in each branch 
    
   - fix: take into account whole parent hierarchy in calculating
     total parallelism for computing variable 
    
   - enhance: support for sending proc_mode config param to custom executors and
     CUSTOM executor config attribute 
    
   - enhance: Torque/PBS: support for sending procs request using nodes/ppn attribute 
     instead of procs 
    
   - fix: 'forward' failing with cryptic errors when attempting to
     forward multiple files 
    
   - enhance: enabled matching of directory name in split pattern when
     directory present in pattern 

   - fix: second and later outputs possibly not cleaned up in some 
     situations of command failure 
    
   - fix: branch variables not accepted as required parameters 

   - fix: output does not include branch if no input as base and output 
     extension syntax used 
    
   - fix: R aborting when user hits control-c in some situations 
    
   - fix: some complex pipelines starting with list failing with errors 
    
   - fix: fixes for PBS Pro executor from Davide 
   
   - enhance: support for PBS Pro thanks to Davide Rambaldi 

=========== Changes for Bpipe Version 0.9.8.6_rc ============

   - support for 'load' to work with URLs for loading 
     scripts remotely

   - fix chr variable created by chr() not considered as satisfying 
     requires chr:...  
     
   - start and stop time not set by local command
     exector 
     
   - restrict use of auto filter to cases where chr actually 
     in operation 
     
   - file() construct sometimes returning paths with .. in them
     
   - Add pipeline stage name to printout in bpipe query 
     
   - support for multiple file extensions when using $output variable
     ($output.foo.bar.gz, etc) 
   
   - fix incorrect outputs resolved for filter when inputs are 
     resolved from a relative path leading with "." character 
     
   - fix preserve not resolving files specified by different absolute /
     relative paths to Bpipe's form, (thx Anthony)
     
   - Not serializable exception on legacy directories 
     
   - Remove legacy feature where output may
     be inferred from created files: does not behave reliably in
     parallelized pipelines 
     
   - support for multiple "preserve" patterns 

   - fix preserved status not displayed
     correctly by bpipe query 
     
   - support for grouping commands in tool database
     into 'modules' 
     
   - fix syntax errors in tool database 
     
   - added igv tools, fix GATK probe, add VEP to tools database 

   - fix binary report files copied as text causing corruption
     
   - add stage name to meta data properties saved for each
     output file 
    
   - fix filter sometimes expecting wrong input when
     file extension supplied 
     
   - fix "bpipe run" leaking a tail process each time on OSX 

   - support for generating reports using arbitrary groovy
     code in a separate script 
     
   - add tools and versions to meta data stored in 
     output properties 
     
   - fix status for SLURM displaying Torque in output 
     
   - fix possible conflicting output names on multiple branches 
     when nested branches cascade without intervening stages 

   - add support for referencing output graph in reports 
     
   - transform should be able to
     match whole file name, fix incorrect input can be resolved
     inside transform() to() construct 
    
   - display running time and pending output in response to 
     bpipe query 
   
   - Add procs to LSF executor 
     
   - refactored a lot of executor stuff to remove duplicated
     code, plus to allow for recording start of execution time
     separate to start of queuing time 

   - fix GB ignored as key for memory 

   - support to use ${output(...)} in commands instead of produce  

   - support for adding jars to classpath via bpipe.config 
     (libs="foo.jar:bar.jar...")

   - fix requires at pipeline stage level no longer working because 
     overriden by top level requires function 

   - experimental: multithread startup to ammeliorate long delays 
     when file system is slow

   - make it possible to remove the cwd option from lsf
     executor (see google group)

   - use -e instead of -eo for LSF since it is compatible with OpenLava 
     
   - ability to use multiple extensions for 'input' variable file 
     extensions, eg: $input.foo.bar.csv returns files ending in 
     '.foo.bar.csv'

   - fix input with file extension not requiring match on first 
     character in some cases 
     
   - fix problem in some weird cases where object has null class 
     
   - prevent errors when running in test mode with explicit 
     notifications sent in pipeline to 'file' channel
     
   - switch to SimpleTemplateEngine because GStringTemplateEngine is not
     observing classpath of added libraries, only core groovy
     libraries 
     
   - fix output directory set by command line
     resulting in inputs not correctly recognised in 2nd and
     later stages of pipeline 
     
   - fix required inputs using exact path instead of file name 
     
  
========== Changes for Bpipe Version 0.9.8.6_beta_3 ============

   - support for setting check message without failing or succeeding 
   - explicitly (check.message = "...." in otherwise clause) 

   - new check report to send / write a summary of all checks 
     as HTML (bpipe run -R checks ..., 
     send report('checks.html') ...)
    
   - documentation now available to reports based on user
     templates 

   - "utils" now provided to templates for generic
     utilities (escape, todo: documentation) 
     
   - Ensure full stack trace is
     logged in bpipe log when missing function encountered 
     
   - make produce() with directory use that directory rather 
     than coerce to default output dir 

   - added new supported commands to default help display 
     
   - fix parameters passed in by command line not resolvable in
     templates 

   - support 'send report(...) to file: <file name>'
     syntax 
     
   - fix subject line used incorrect in SMTP notifications
     
   - inputs not being correctly checked to be provided when specified 
     at root level 

   - don't display exception info when failure is due to
     value missing from specified inputs via 'inputs ...' statement
     
   - support for 'requires' at top level outside of pipeline 
     definition 
     
   - improve wording and display of error when required value is
     missing 

   - fix templates not able to see all classes available on
     classpath

   - removed all forms of throttling of R sessions (should now be
     unnecessary) as it can form a bottleneck in certain
     scenarios 


========== Changes for Bpipe Version 0.9.8.6_beta_2 ============

   - avoid warnings due to project properties (thanks Davide) 

   - refactored email functionality to use user configurable
     templates 

   - fix notifications for pipeline finished saying
     pipeline failed when it didn't 

   - support for displaying information about checks at end of pipeline 

   - more information in SMTP notifications, including checks 
     
   - Added option to override all checks for cases where there are 
     very many that have failed 
     
   - changed unknown communication provider to be a warning instead 
     of a pipeline terminating exception.

   - fix relative symbolic link to Bpipe not resolving properly 
     under Mac/OSX 

   - fix possibility of commands sometimes
     not being cleaned up (add pause before terminating Bpipe,
     after terminating child processes) 

   - Fix Slurm not returning failure exit code when
     job cancelled via bpipe stop (causing outputs not to be
     cleaned up)

   - Added "checks" command and "override" command to display and 
     override checks respectively

   - Fix transforms sometimes applying in such a way
     as to add branch name with no period separating them from
     the rest of the file name (regression)
     
   - fix transforms producing null outputs when no period in 
     input file name  (regression)
     
   - support for rendering pipeline diagrams as SVG 

   - support for specifying pipeline stages to ignore 
     in diagrams via noDiagram="..." in bpipe.config (annotations to 
     come in the future) 

   - fix diagrams generated incorrectly if 'segment'
     command used 

   - fix documentation failing to generate unless script has a
     relative or absolute path prefixing it 

   - some tweaks to avoid rare cases double
     dots appearing in file names with transform 

   - 'succeed' should terminate parent branch when all child
     branches terminated via succeed 

   - support for running a pipeline "until" a specified stage
     ( -u)  

   - support for send to use HTML templates instead of
     inline HTML 
     
   - support for 'succeed' and 'fail' to share
     'send' syntax to generate an email or IM along with the
     termination of the branch 

   - fix slurm executor sometimes
     treating slurm stdout and stderr temp files 
     as real outputs of pipeline 

   - Fix space inserted when withFlag used with flags 
     ending with "=" 
     
   - fix variables and functions loaded via 'load' not available until 
     pipeline launched (inside run { } ) 
     
   - allow global resource limits (as supported by -l flag 
     to bpipe) to be specified in bpipe.config as well 
   - support for sending emails and
     instant message notifications explicitly 
     from script, using syntax: send text 'hello' to gtalk, 
     and more advanced forms (documentation to come) 

   - support for using
     explicit regex pattern to split files to parallel 
     paths (as opposed to simplified pattern syntax) 
     just add ~ to pattern 
     
   - fix (very rare) possibility for duplicate file
     names when referencing 'inputs' 
     variable 
     
   - added support for matching numbers
     in input splitting patterns ('#') 

   - added support for gradle wrapper 
     
   - added gradle wrapper for easier building
     (now just use gradlew instead - of gradle, and it
     self downloads) 
     
   - fix custom resources not enforced when
     provided as map 

   - fix default build to autodetect Gridgain
     jars and compile without if not 
     found 

   - get mail.jar from Maven so default checkout compiles 
     
   - changes to allow "single" queue to handle
     multiple tasks (vlsci) 

   - fix 'bpipe status' displaying incorrect command for SLURM, PBS
     possibly others
     
========== Changes for Bpipe Version 0.9.8.6_beta_1 ============

   - fixed file(...) function sometimes returning file object
     for "<file>/." 

   - experimental feature: supply closure with ".using(...)" for lazy 
     evaluation, allowing reference to runtime pipeline variables in 
     configuration of pipeline stages

   - initial implementation of a 'bpipe status' command, to show exactly
     what is currently running 

   - Ability to explicitly cause a
     branch to either fail or succeed prematurely 
     (via "fail" and "succeed" commands) 

   - added support to transform a
     single file pattern to multiple targets 

   - ability to configure and customize reports from bpipe.config 
     (custom names, etc.), and to send by email when email notifications
     configured 

   - allow "procs" to be specified as integer in
     resource manager config 

   - fix first line of log file sometimes not showing when using 
     "bpipe log ..." 

   - "bpipe query" not working on files in non default output
      directories (eg: specified with output.dir=....) 

   - more graceful / better handling of errors
     that cause the script to not parse or load at all 

   - support for finding bpipe.config in the same directory as
     the pipeline script (which might be different to the
     directory that bpipe is running in) 

   - better handling of exception when probing tools in tool database 

   - new feature: allow for muliply by Map to set an explicit
     mapping of branch names to inputs 

   - special behavior for accompanies when dealing with 
     transforms: always adds
     extension to end of file rather than replacing 

   - allow requires to specify complete file name instead of just
     extension 

   - allow resolution of 'load' files relative to the pipeline
     script itself so that it is easy to maintain a pipeline
     file and ancillary files in the same directory 

========== Changes for Bpipe Version 0.9.8.5 ============

-   fix output.dir not working if transform annotation used 

-   fix silly mistake causing load external stages to fail 

-   fix loading a pipeline stage from within a loaded pipeline stage not
    working

-   added ability to set branch level variables that are carried forward
    through the pipeline via branch.<name> = "something" syntax

-   Fix output sometimes not fully flushed when running pipelines using
    resource managers

-   support for using arbitrary lists for parallelization over any  
    list of items

-   fix msg writing output during probes (causing double output in 
    certain situations)

-   when an output that is specified as an absolute / relative file that exists, 
    fix not seeing it in dependency graph

-   better file names when output extension occurs in combination with 
    introduction of branch name (foo.chr1.csv instead of foo.txt.chr1.csv)
    NOTE: in rare situations this might cause output file names to change,
    thus may cause outputs to get recomputed for existing pipelines (in rare cases)

-   allow command line region to override chr() within pipeline

-   fix command configuration sometimes not displaying name 

-   slightly better description of command for specifying report 
    template (see Google Group discussion)

-   fix R code failing when last line contains certain syntax

-   Enable specifying file name for reports (Patch from Ido Tamir,
    thank you!)

-   fix problems when multiple implicit outputs computed from same input file 
    (output1.foo and output2.foo end up the same file)

-   handle incorrect report template much more nicely than a giant stack
    trace

========== Changes for Bpipe Version 0.9.8.4.1 ============

-   fix middle parallel section causing spurious error if no inputs provided
    at start of pipeline

-   fix error when pipeline starts with parallel section and adds a
    subsequent stage (regression)

-   fix file splitting patterns not working properly for very specific case
    of a split wildcard (%) followed by 2 chars


========== Changes for Bpipe Version 0.9.8.4 ============

-    fix __bpipe_lazy_resource_threads__ appearing in
     output when commands fail instead of real value of $threads

-    fix __bpipe_lazy_resource_threads__ appearing in
     command log instead of real value of $thread

-    prevented tools being probed more often than needed

-    moved throttling of command execution into separate
     class, applied throttling to tool probing so as to lessen chances
     of exhausting threads or file handles when running with 
     large numbers of inputs and highly parallel pipelines

-    fix log.warn causing any UNKNOWN status code return
     from PBS/Torque to exit with error instead of retrying

-    fix filter transforms sometimes basing output file
     on wrong input 

-    preliminary SLURM support via Andrew Lonsdale (thanks!)

-    when a command is configured with to use a specific
     number of procs, that should override the magic $threads
     variable value

-    PBS / Torque support not sending memory config for
     queues other than 'batch' or 'smp'

-    hack: add ability to put extra classes into
     $BPIPE_LIB/extra-lib.jar for use in pipeline script

-    don't convert outputs containing paths to output
     folder

-    further support for inputs that are directories -
     prevent checking of timestamp of directory, search the 
     full output tree instead of just immediate inputs to stage 
     when a directory input is requested

-    fix glob outputs not recognized as already created
     when in explicitly set output folder

-    fix "using" construct causing diagrams to fail

-    fixed diagram failure on nested parallel section, and
     when a parallel section is the root element in the pipeline

-    fix diagrams failing with exception stack trace when
     pipeline is split by chromosome

-    fix diagrams not working when only a single stage
     defined in pipeline

-    add support for transform(".xml") to(".csv") { }
     style syntax (experimental)

-    better error message for referencing undefined function

-    fixed branch names including directory portions of
     input file paths when split on files from other directory

-    ensure variables can be overriden from the command
     line

-    show all commands when test mode aborts a multi
     command

-    option to show only output from single thread with
     bpipe log -t


========== Changes for Bpipe Version 0.9.8.3 ============


-    fix input not resolved by from / input ext when
     reference embedded in nested

-    fix %bin% expansion incorrect when tool command is
     part of the full path containing the tool itself - eg:
     /usr/local/bwa/bwa-0.5.9/bwa

-    fix possible double period in output file name (regression)

-    fix references to output.dir causing outputs to be
     inferred even when no outputs actually decided yet for
     stage (eg: before produce / transform / filter, etc.).

-    fix backslashes in file paths causing invalid output
     paths on Windows

-    fixed failures when files without extensions created
     as outputs / inputs

-    run commands using -e option with bash so that failure
     of any command within script results in abort

-    nicer error message when a filter encounters the
     situation that no inputs were provided

-    fix transform producing same output as input when
     output file of same type as input

-    add support for closure as value for variable default
     to compute default value at runtime, prevent referencing  
     unless actually needed (useful eg: if the default should be 
     an input when not provided)

-    attempt to fix out-of-order execution of pipelines
     that start with list and append stages after

-    support for "requires" to specify required parameters
     for a pipeline stage

-    support for multiple paths in BPIPE_LIB

-    support for script files in BPIPE_LIB (in addition
     to folders of scripts)

-    support for ordering of execution of modules in
     BPIPE_LIB

-    fix output files not merged correctly when multiple
     levels of nested
     parallel segments end at same point

-    updated to add withFlag for more stringent test case

-    changed build to use manually downloaded gridgain lib
     (GridGain pulled their repo support :-( )


========== Changes for Bpipe Version 0.9.8.1 ============

-  fix double spaced output

-  fix input.size() and inputs.size() throwing exceptions

-  fix potential incorrect output inferred in filter operation,
   causing 'expected output not found' error in some filter 
   scenarios

========== Changes for Bpipe Version 0.9.8.1 ============

-    Fix race condition causing occasional null pointer
     exception, making param_nested_parallel test fail

-    fix exception when using bpipe preserve

-    fix potential leading semicolon when executing command

-    fix for incorrect input inferred by filter if actual
     input used different to those supplied by previous stage

-    fixed regression causing '..' in filenames from output
     extension that operates like transform

-    fix double period in inferred output names when no
     explicit filter / transform present

-    fixed dependencies working incorrectly with produce; stage skipped even if
     actual inferred / used input was newer than actual inferred output

-    fix multiple instances of Bpipe running R scripts
     clashing other

-    support for "diagramOrientation="vertical" in
     bpipe.config to change orientation of diagrams

-    fix diagrams not generating correctly for some
     pipeline architectures

-    fix commands used to generate outputs sometimes
     displaying as "..." when using "bpipe query"

-    improvements to tool database - more links, better
     descriptions, etc.

-    to prevent some of the massive output 
     that comes from large pipelines when re-executing, buffer
     messages unless a stage actually does something.

-    support for producing custom reports, named by
     command line

-    added source for software schedule report

-    fix input ext on output with index referencing
     wrong output

-    fix cleanup not removing directory

-    fix glob on output folder not working

-    fixed output explicitly specified in produce() to be
     in folder still remapped to default output dir

-    added alternative bpipe runner that runs in background

-    fixed filterInputs still not working right with chr()
     behavior now defaults based on whether the input files contain
     chromosome references rather than always filtering


========== Changes for Bpipe Version 0.9.8 ============
 
-   fixed problems with quoted args becoming unquoted during launch
    (so eg: bpipe execute 'foo + bar' would not work)

-   fixed 'bpipe retry' not working with quoted arguments
    in commands

-   fixed filter sometimes incorrectly refusing to allow output files that
    should work basd on used inputs

-  fix $input1 not resolving in commands if there is only one input

-  fix no information about outputs printed when too many outputs

-  added 'withFlag' function on $inputs, $inputs.<ext> variables for easily
   adding multiple inputs with the same flag (eg: ${inputs.bam.withFlag("-I")})

-  strip leading spaces from commands when joining lines together for execution
   (prevent whitespace bloat in command log)

-  jobid now prepended in history file

-  support to retry specified job from history using
   bpipe retry <jobid>

-  prevent bpipe commands that fail with invalid
   arguments from getting logged in history

-  don't log full path to bpipe executable in history
   file (breaks if bpipe location / version changes)

-   fixed files failing to get cleaned up if same file was
    previously cleaned up + added warning if cleanup fails

-   fixed filter followed by from potentially resolving
    its own stage output as input

-   fix for mt issue around pipeline inputs (causing
    parallel_same_stage_outputs test to fail if you run
    it many times) - regression from beta 3/4

-   support for @intermediate and @accompanies
    (experimental) annotations

-   fixed failure to cleanup outputs that are inferred from output file
    extensions rather than transform / filter / produce when pipeline 
    stage fails

-   fixed multiple references to multiinput properties
    resulting in duplicated / missed outputs

-   fixed reference to inputs variable followed by
    reference to input variable returning wrong results

-   fixed later stages not resolving inputs from non-final
    stages in prior nested parallel segments

=========== Changes for Bpipe Version 0.9.8_beta_5 ============

-   more fixes to make config{} work properly
    fixed 'config' not working / failing to execute
    commands
-   fix parallel branches created from file pattern split
    not using split files as basis for outputs sometimes

=========== Changes for Bpipe Version 0.9.8_beta_4 ============

-   support for multiple split characters in input matching patterns

-   support for config closure to enable config across multiple commands or
    when using "multi"

-   slightly better messaging when skipping command due to existence, not
    based on time stamp

-   fix display of output to stderr when version probe command fails
    - send to log instead

-   experimental chaining of from and filter / transform

-   fix "from" causing original inputs not to be queried for files in some
    situations

-   fix "filter" nested in "from" failing to infer correct output when output 
    based on input other than input in "from" clause

-   fix possible exception when using $output.prefix

-   fixed second and later stages in nested parallel segments not always
    resolving correct inputs

-   fix attempts to use "split" in input variables failing (regression)

-   added support for use of "region" variable without genome or chr splitting
    to allow generic pipeline stages that can work with or without regions 
    specified
    
-   added command line option ("-L") to specify default value of region variable
    
-   fixed "from" to match multiple inputs if used in form from("*.txt") etc.

-   Added example for how to run paired end reads

-   fix output extension causing wrong / confusing behavior when incombination
    with a filter and unexpected input order

-   fix bpipe command leaving out eclipse folder from classpath

=========== Changes for Bpipe Version 0.9.8_beta_3 ============

-   Updated underlying groovy implementation to Groovy 2.1

-   Add "pre-existing" label to display of outputs that 
    weren't genereated by the current run

-   fix error when creating File objects from input /
    output

-   support for using numbered outputs with file extensions
    (eg: $output2.txt)

-   fixed outputs sometimes being recreated unnecessarily
    when multiple commands inside transform/filter

-   support for implicit 'thread' variable for auto
    allocating threads based on known concurrency

-   fixed duplicate output files for $output1,
    $output2,... if pipeline has no input files

-   fix jobs not always cleaned up from global job table

-   support for referencing a particular input with a
    particular extension using "$input<n>.<ext>"

-   fixed double period appearing in some file names when
    using filter

-   fixed order of output from "glob" depending on the
    vagaries of the file system or java internals by
    sorting it

-   made 'from' conform to the convention of multiple
    inputs of same time bumping to next actual input
    instead of returning the same over and over

=========== Changes for Bpipe Version 0.9.8_beta_2 ============

-   fixed multi not failing pipeline when an asynchronous
    command fails

-   making filter() conform to the same behavior as
    transform(): when a filter type is repeated, it moves
    to the next input (see google group discussion -
    https://groups.google.com/forum/?fromgroups=#!topic/bpipe-discuss/G58ExlG2Vgo

-   fixed error when reporting notification failures
    to user

-   fixed output extensions not working correctly with
    filter when multiple filters supplied

-   fix multi not actually parallelizing properly in
    some cases

-   fix regression in 'from' not accepting list (added
    test)

-   fixed from with multiple arguments ignoring first
    argument

-   fixed deadlock condition in using "multi" in certain
    cases

=========== Changes for Bpipe Version 0.9.8_beta_1 ============

-   More flexible model for concurrency management, enabling
    management of any kind of resource. Introduction of
    new keyword "uses" to declare resource usage for a block
    and command line options (-m, -l, -n) to control different
    aspects of concurrency

-   Complete re-implementation of dependency analysis so that
    Bpipe understands the depedency graph and can perform
    transitive dependency analysis to allow rerunning piplines
    without rebuilding unnecessary intermediate dependencies.
    (See Issue #2, http://code.google.com/p/bpipe/issues/detail?id=2)
    Includes support for new command line commands, 
    "bpipe query", "bpipe cleanup", and "bpipe preserve" to 
    display and manage dependencies, as well as annotations,
    "@preserved" to declare dependencies that should not be
    deleted in a "cleanup". 

-   support for new 'glob' function available within
    pipeline stages to facilitate pipelines that need to
    use wild card matches on input files.

-   Annotations can be used with lower case now. Upper case 
    still works, but some people feel @transform, @filter,
    etc. looks nicer.

-   Support for parallelizing by chromosome on existing
    files already split by chromosome. If your input files
    are split and named by chromosome, you can use 
    filterInputs parameter to make the existing
    files be used for inputs to the parallelization 
    eg. chr(1..2, filterInputs:true) 

-   Many various bug fixes and improvements to messaging,
    behaviour and output.