Recently, I have blogged about Automatic Differentiation (AD) and my motives for developing an AD library to experiment on it. Even though I have recently made it public, I have been investigating other Julia packages about AD, especially how they define the differentiation rules. If a Julia package isn’t designed for a very specific task, then it probably consists of multiple sub-packages that support its functionality. Julia language has an ecosystem where packages are designed as multiple sub-packages to handle part of the requirements. This has two positive effects of the Julia package ecosystem; one may find a package for various needs, every project contributes to this ecosystem with its sub-packages which is good for the open-source ecosystem (There are nice tips and guides on the Julia Blog).

What is my point? The point is some of the earliest AD libraries of Julia are using their own set of differentiation rules. They are also providing ways to define custom differentiation rules and avoid automatic differentiation process. It is good, however, knowing that the behavior of the AD process may change every time one switches a library isn’t very intuitive. On the other hand, it is not easy to integrate a library to work with another. Because the focus of the libraries vary; they may be focusing on limiting features and functionality to gain speed.

Since I have learned more about Julia’s ecosystem, I have started digging into collections maintained by Organizations on GitHub. JuliaDiff is one of them. It is really nice to be able to find most of the packages developed for a purpose, under the same roof. There are multiple Julia packages for AD. There are also very well documented packages that do AD tasks. However, they seem to opt-out modularity. I have been looking into DiffRules.jl for a while. They have a very nice set of differentiation rules, also ways to define custom rules. However, most importantly they return symbolic derivations for operations. This makes things extendable… Some of the AD packages I have mentioned so far(without giving names) also modular(in their own way…) but not using already existing DiffRules.jl package, which is only causing the same differentiation-rules to be re-defined again every time someone develops a new library(for example; I couldn’t use a very popular one because it wasn’t extendable). I believe DiffRules is generic enough, since it return derivative function symbolically, any library may have built their own upon them or change them on package initialization(like I did this time). However, most did not…

As I have published AutoDiff.jl library, I have also re-defined a small subset of functions(Because this was just an experimental project on mine). However, in order to be functional, more functions needed to be defined. As one can guess I did not define them by hand… Because there is DiffRules.jl. Instead, I have followed some kind of a meta-programming approach. I was able to generate AutoDiff.jl compatible rules out of every rule defined in the DiffRules.jl. This means now, AutoDiff.jl potentially much more capable(as much as DiffRules.jl allows at least). An interesting thing is that only 64 lines of code were enough for it to maintain its existing functionality and expand on it…

The latest changes are merged into the master branch since I have not observed any loss of functionality.