Setting up Neanderthal with REBL and Cursive

In this post, I'll demonstrate how to set up Cognitect's REBL to run nicely in conjunction with Neanderthal. I have been finding working with REBL an absolute pleasure and I believe it could add a lot to the Clojure ecosystem if it gets some momentum.

The following versions of components will be used:

Java 11
IntelliJ IDEA 2019.1.3
Cursive 1.8.2
Intel MKL 2019.4
Neanderthal 0.25.3
REBL 0.9.172

setup

I will only mention the complex parts of the configuration as for the most you can follow these three guides to get the majority of the relevant setup done:

Neanderthal Getting Started Guide

REBL Java 11

REBL Cursive

MKL mklvars

I usually start all my applications via TMux sessions and I include the initialization of MKL in my .zshrc. Below the CUDA LD path isn't necessary for this post but I've included it as I use CUDA as well.

# -------------------------------
#    Neanderthal, LD Libraries
# -------------------------------

export CUDA_LD=/usr/local/cuda-10.1/lib64
export LD_LIBRARY_PATH=$CUDA_LD
. /opt/intel/compilers_and_libraries_2019.4.243/linux/mkl/bin/mklvars.sh intel64

datafyzshrc.zsh — view on GitHub

project layout

I include the REBL jar under my resources path as I sometimes use its API from my user.clj. I will demonstrate this more below.

λ ubuntu scorpion → λ git master → tree
.
├── CHANGELOG.md
├── deps.edn
├── README.md
├── replmem.sh
├── resources
│   └── REBL-0.9.172.jar
├── src
│   ├── nz
│   │   └── co
│   │       └── arachnid
│   │           └── scorpion
│   │               ├── core.clj
│   │               ├── matrixcuda.clj
│   │               ├── matrixnative.clj
│   │               ├── matrixopencl.clj
│   │               └── repl
│   │                   ├── cuda.clj
│   │                   └── opencl.clj
│   └── user.clj

datafyneanderthal1.sh — view on GitHub

deps

Note the REBL jar is included with my core deps as opposed to with the rebl alias so I can use the REBL API. We also need to add some additional jvm-opts to get Neanderthal to work with JDK 11.

{:deps {org.clojure/clojure          {:mvn/version "1.10.1"}
        uncomplicate/neanderthal     {:mvn/version "0.25.3"}
        org.clojure/java.classpath   {:mvn/version "0.3.0"}
        org.clojure/tools.namespace  {:mvn/version "0.2.10"}
        com.cognitect/rebl           {:local/root "resources/REBL-0.9.172.jar"}}
 :aliases
       {:rebl
        {:extra-deps
                     {org.clojure/core.async      {:mvn/version "0.4.490"}
                      org.openjfx/javafx-fxml     {:mvn/version "11.0.1"}
                      org.openjfx/javafx-controls {:mvn/version "11.0.1"}
                      org.openjfx/javafx-swing    {:mvn/version "11.0.1"}
                      org.openjfx/javafx-base     {:mvn/version "11.0.1"}
                      org.openjfx/javafx-web      {:mvn/version "11.0.1"}}
         :jvm-opts   ["--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED"]
         :main-opts  ["-m" "cognitect.rebl"]}}}

datafyneanderthal2.edn — view on GitHub

cursive run config

datafy neanderthal

Neanderthal makes heavy use of Protocols and Types which I really like. Two of the implemented Protocols expose functionality that works well with REBLs Datafiable protocol. These allow us to easily expose the contents of a Matrix and its associated metadata.

(ns nz.co.arachnid.scorpion.matrixnative
  (:use [uncomplicate.neanderthal core native linalg])
  (:use [uncomplicate.commons.core])
  (:require [clojure.core.protocols :as p])
  (:import (uncomplicate.neanderthal.internal.host.buffer_block RealGEMatrix)))

;; ===================================
;; Patch in additional datafy support
;; ===================================

(extend-protocol p/Datafiable
  RealGEMatrix
  (datafy [x] (with-meta
                (seq x)
                (info x))))

datafyexprotocol.clj — view on GitHub

stepping through expressions

Now when I evaluate a series of Neanderthal expressions in Cursive the expressions are automagically piped off to REBL as well. The Matrix types are now also understood by REBL and rendered nicely along with their metadata.

;; =========================
;;  Solving a Linear System
;; =========================

(def coeffecient-matrix (dge 3 3 [1 -1 1
                                  4  1 0
                                  0  1 4] {:layout :row}))

(def resulting-matrix (dge 3 1 [0
                                8
                                16] {:layout :row}))

(defn solve-linear-system
  [coeffecient-matrix result-matrix]
  (sv! coeffecient-matrix result-matrix))

(solve-linear-system coeffecient-matrix resulting-matrix)

datafylinearsystem.clj — view on GitHub

cursive

REBL expressions pane

REBL data pane

Now the metadata we exposed via the extended protocol is displayed in the top panel. The Matrix data is displayed in the lower window.

using the REBL API

Before running REBL directly via a Cursive config I would fire up REBL through my REPL session and send expressions off to REBL on demand. This is a good solution if you prefer your standard editors REPL for most of your needs and then just use REBL for exploring larger datasets.

(ns user
  (:require [clojure.pprint :refer :all]
            [clojure.repl :refer :all]
            [clojure.tools.namespace.repl :refer :all]
            [cognitect.rebl :as rebl]))

(defn list-namespaces
  "Lists all the namespaces on the classpath"
  []
  (sort
    (map
      (fn [namespace] (.getName namespace))
      (all-ns))))

(defn list-public-symbols
  "Lists all the public symbols for a namespace
   Should be called with a symbol argument.

   ```
   (list-public-symbols 'taoensso.timbre)
   ```"
  [sym-namespace]
  (sort (ns-publics (find-ns sym-namespace))))

(defn print-map-table
  [col-m]
  (print-table col-m))

(defn repl-reload
  []
  (refresh))

(defn rebl-start
  "Start a new REBL UI"
  []
  (rebl/ui))

(defn rebl-send
  "Pipe off the given expression to a running REBL session."
  [expression]
  (rebl/inspect expression))

datafyuser.clj — view on GitHub