The RASP programming language by Weiss et al. is a computational model of transformers and used to reason about this sequence-to-sequence deep learning architecture. Since (symbolic) RASP and (neural) transformer-encoders are two sides of the same coin, it seems interesting to combine them to create a neuro-symbolic language, in which the symbolic part of a program can be designed and written by programmers, while the neural parts are learned by machines. As a first step towards using RASP, we provide an alternative implementation of RASP. While vanilla RASP is implemented in Python, our implementation is done in Scala. To embed RASP into Scala, we extend RASP with a type system and document its implementation. With our alternative RASP implementation, we discover and fix a few bugs in the official RASP test suite. On the fixed test suite, ScalaRASP achieves 100 % correctness and a speedup compared to vanilla RASP by one to two orders of magnitude (owed to the performance difference between the JVM and CPython). Then, we take another step towards a neuro-symbolic language, by adding transformer-encoders to the RASP language. To integrate the neural and the symbolic components of a program, we implement a translation from neuro-symbolic programs to a common intermediate representation, i.e., to computation graphs which are end-to-end differentiable. Our computation graph translation also passes the RASP test suite. Using automatic differentiation and differentiable relaxations, we facilitate end-to-end training of integrated transformer-encoders, i.e., jointly within the containing RASP program, rather than separately. The training of neuro-symbolic ScalaRASP programs is left for future work.
Daniel Maninger (Wed,) studied this question.