Code Generation

Code Generation

Drill relies heavily on code generation for the data-specific functionality in each operator. For example, the FilterBatch (implements the SQL WHERE clause) generates code for the actual filter condition. Code generation is necessary because Drill works with a large number of data types (over 120 different value vector implementations) and many kinds of expressions. The alternative, an interpreter, would be expensive to maintain (given the large number of value vector types) and slower to execute.

Drill has two major forms of code generation:

1. The FreeMarker-based code generation done during the build process, and
2. The JCode-based code generation done at execution time.

This writeup focuses on the second form.

(Code generation seems to follow a pattern established by Hive? Need to research.)

Topics

Code generation is a complex topic covered over multiple pages:

• Code Generation Workflow
• Components
• Code Cache
• Code Transformations
• Scalar Replacement

References

Compilers:

• JavaCompiler in the JDK.
• javax.tools explanation.
• Janino Compiler

Byte code manipulation:

• ASM Package
• ASM Tutorial
• Merging Class Methods with ASM
• Scalar replacement research paper

Compilation Framework

Background

Java compilation in Java itself is based on the JavaCompiler class. The javax.tools provides a dynamic compilation framework. The IBM article provides a sample app that compiles Java classes from in-memory sources.

Debugging Hints

You can view the generated code in one of three ways:

• Enable debug logging for AbstractClassCompiler. The generated code is written to the log file. Set the config option drill.exec.compile.debug to true if you want line numbers added to each line.
• Uncomment the obvious block of code in AbstractClassCompiler to write each generated class to /tmp. This version is handy as it uses the class name to name each file.
• Pass the following to the command line when starting Drill: -Dorg.codehaus.janino.source_debugging.enable=true -Dorg.codehaus.janino.source_debugging.dir=/tmp. (See this post for more information.) This version writes all files using generic temporary names, making it hard to find the particular file of interest.

You can gain access to the generated class files by uncommenting the obvious lines near line 256 in MergeAdapter.