The Sense of Proprioception


Written by:

Robert McIntyre

1 Proprioception

Close your eyes, and touch your nose with your right index finger. How did you do it? You could not see your hand, and neither your hand nor your nose could use the sense of touch to guide the path of your hand. There are no sound cues, and Taste and Smell certainly don't provide any help. You know where your hand is without your other senses because of Proprioception.

Humans can sometimes loose this sense through viral infections or damage to the spinal cord or brain, and when they do, they loose the ability to control their own bodies without looking directly at the parts they want to move. In The Man Who Mistook His Wife for a Hat, a woman named Christina looses this sense and has to learn how to move by carefully watching her arms and legs. She describes proprioception as the "eyes of the body, the way the body sees itself".

Proprioception in humans is mediated by joint capsules, muscle spindles, and the Golgi tendon organs. These measure the relative positions of each body part by monitoring muscle strain and length.

It's clear that this is a vital sense for fluid, graceful movement. It's also particularly easy to implement in jMonkeyEngine.

My simulated proprioception calculates the relative angles of each joint from the rest position defined in the blender file. This simulates the muscle-spindles and joint capsules. I will deal with Golgi tendon organs, which calculate muscle strain, in the next post.

2 Helper Functions

absolute-angle calculates the angle between two vectors, relative to a third axis vector. This angle is the number of radians you have to move counterclockwise around the axis vector to get from the first to the second vector. It is not commutative like a normal dot-product angle is.

(in-ns 'cortex.proprioception)

(defn right-handed?
  "true iff the three vectors form a right handed coordinate
   system. The three vectors do not have to be normalized or
  [vec1 vec2 vec3]
  (pos? (.dot (.cross vec1 vec2) vec3)))

(defn absolute-angle
  "The angle between 'vec1 and 'vec2 around 'axis. In the range 
   [0 (* 2 Math/PI)]."
  [vec1 vec2 axis]
  (let [angle (.angleBetween vec1 vec2)]
    (if (right-handed? vec1 vec2 axis)
      angle (- (* 2 Math/PI) angle))))
(in-ns 'cortex.proprioception)
(absolute-angle  Vector3f/UNIT_X Vector3f/UNIT_Y Vector3f/UNIT_Z)
(in-ns 'cortex.proprioception)
 Vector3f/UNIT_X (.mult Vector3f/UNIT_Y (float -1)) Vector3f/UNIT_Z)

3 Proprioception Kernel

Given a joint, proprioception-kernel produces a function that calculates the Euler angles between the the objects the joint connects.

(defn proprioception-kernel
  "Returns a function which returns proprioceptive sensory data when
  called inside a running simulation."
  [#^Node parts #^Node joint]
  (let [[obj-a obj-b] (joint-targets parts joint)
        joint-rot (.getWorldRotation joint)
        x0 (.mult joint-rot Vector3f/UNIT_X)
        y0 (.mult joint-rot Vector3f/UNIT_Y)
        z0 (.mult joint-rot Vector3f/UNIT_Z)]
    (fn []
      (let [rot-a (.clone (.getWorldRotation obj-a))
            rot-b (.clone (.getWorldRotation obj-b))
            x (.mult rot-a x0)
            y (.mult rot-a y0)
            z (.mult rot-a z0)

            X (.mult rot-b x0)
            Y (.mult rot-b y0)
            Z (.mult rot-b z0)
            heading  (Math/atan2 (.dot X z) (.dot X x))
            pitch  (Math/atan2 (.dot X y) (.dot X x))

            ;; rotate x-vector back to origin
            (doto (Quaternion.)
               (.angleBetween X x)
               (let [cross (.normalize (.cross X x))]
                 (if (= 0 (.length cross)) y cross))))
            roll (absolute-angle (.mult reverse Y) y x)]
        [heading pitch roll]))))

(defn proprioception!
  "Endow the creature with the sense of proprioception. Returns a
   sequence of functions, one for each child of the \"joints\" node in
   the creature, which each report proprioceptive information about
   that joint."
  [#^Node creature]
  ;; extract the body's joints
  (let [senses (map (partial proprioception-kernel creature)
                    (joints creature))]
    (fn []
      (map #(%) senses))))

proprioception! maps proprioception-kernel across all the joints of the creature. It uses the same list of joints that cortex.body/joints uses.

4 Visualizing Proprioception

Proprioception has the lowest bandwidth of all the senses so far, and it doesn't lend itself as readily to visual representation like vision, hearing, or touch. This visualization code creates a "gauge" to view each of the three relative angles along a circle.

(in-ns 'cortex.proprioception)

(defn draw-sprite [image sprite x y color ]
   (for [[u v] sprite]
     (.setRGB image (+ u x) (+ v y) color))))

(defn view-angle
  "create a debug view of an angle"
  (let [image (BufferedImage. 50 50 BufferedImage/TYPE_INT_RGB)
        previous (atom [25 25])
        sprite [[0 0] [0 1]
                [0 -1] [-1 0] [1 0]]] 
    (fn [angle]
      (let [angle (float angle)]
        (let [position
              [(+ 25 (int (* 20 (Math/cos angle))))
               (+ 25 (int (* -20 (Math/sin angle))))]]
          (draw-sprite image sprite (@previous 0) (@previous 1) 0x000000)
          (draw-sprite image sprite (position 0) (position 1) color)
          (reset! previous position))

(defn proprioception-display-kernel
  "Display proprioception angles in a BufferedImage"
  [[h p r]]
  (let [image (BufferedImage. 50 50 BufferedImage/TYPE_INT_RGB)
        previous-heading (atom [25 25])
        previous-pitch   (atom [25 25])
        previous-roll    (atom [25 25])
        heading-sprite [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
        pitch-sprite   [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
        roll-sprite    [[0 0] [0 1] [0 -1] [-1 0] [1 0]]
        (fn [angle sprite previous color]
          (let [angle (float angle)]
            (let [position
                  [(+ 25 (int (* 20 (Math/cos angle))))
                   (+ 25 (int (* -20 (Math/sin angle))))]]
              (draw-sprite image sprite (@previous 0) (@previous 1) 0x000000)
              (draw-sprite image sprite (position 0) (position 1) color)
              (reset! previous position))
    (dorun (map draw-angle
                [h p r]
                [heading-sprite pitch-sprite roll-sprite]
                [previous-heading previous-pitch previous-roll]
                [0xFF0000 0x00FF00 0xFFFFFF]))
(defn view-proprioception
  "Creates a function which accepts a list of proprioceptive data and
   display each element of the list to the screen as an image."
  (view-sense proprioception-display-kernel))

5 Proprioception Test

This test does not use the worm, but instead uses two bars, bound together by a point2point joint. One bar is fixed, and I control the other bar from the keyboard.

(in-ns 'cortex.test.proprioception)

(defn test-proprioception
  "Testing proprioception:
   You should see two floating bars, and a display of pitch, yaw, and
   roll. The white dot measures pitch (spin around the long axis), the
   green dot measures yaw (in this case, rotation around a circle
   perpendicular to your line of view), and the red dot measures
   roll (rotation around a circle perlendicular to the the other two

     r : rotate along long axis
     t : opposite direction of rotation as <r>
     f : rotate in field of view
     g : opposite direction of rotation as <f>

     v : rotate in final direction
     b : opposite direction of rotation as <v>"

  ([] (test-proprioception false))
     (let [hand    (box 0.2 1 0.2 :position (Vector3f. 0 0 0)
                        :mass 0 :color ColorRGBA/Gray :name "hand")
           finger (box 0.2 1 0.2 :position (Vector3f. 0 2.4 0)
                       :mass 1
                       (ColorRGBA. (/ 184 255) (/ 127 255) (/ 201 255) 1) 
                       :name "finger")
           joint-node (box 0.1 0.05 0.05 :color ColorRGBA/Yellow
                           :position (Vector3f. 0 1.2 0)
                           :rotation (doto (Quaternion.)
                                        (/ Math/PI 2)
                                        (Vector3f. 0 0 1)))
                           :physical? false)
           creature (nodify [hand finger joint-node])
           finger-control (.getControl finger RigidBodyControl)
           hand-control (.getControl hand RigidBodyControl)
           joint (joint-dispatch {:type :point} hand-control finger-control
                                 (Vector3f. 0 1.2  0)
                                 (Vector3f. 0 -1.2 0) nil)

           root (nodify [creature])
           prop (proprioception-kernel creature joint-node)
           prop-view (view-proprioception)]
        (.getControl hand RigidBodyControl)
          ["key-r" "key-t" "key-f" "key-g" "key-v" "key-b"]
          [1 1 10 10 10 10]
           (fn [world]
             (let [timer (RatchetTimer. 60)]
               (.setTimer world timer)
               (display-dilated-time world timer))
             (if record?
                (File. "/home/r/proj/cortex/render/proprio/main-view")))
             (set-gravity world (Vector3f. 0 0 0))
             (enable-debug world)
             (light-up-everything world))
           (fn [_ _]
              (list (prop))
              (if record?
                (File. "/home/r/proj/cortex/render/proprio/proprio"))))])))))

6 Video of Proprioception


Proprioception in a simple creature. The proprioceptive readout is in the upper left corner of the screen.

6.1 Generating the Proprioception Video


(defn images [path]
  (sort (rest (file-seq (File. path)))))

(def base "/home/r/proj/cortex/render/proprio/")

(defn pics [file]
  (images (str base file)))

(defn combine-images []
  (let [main-view (pics "main-view")
        proprioception (pics "proprio/0")
        targets (map
                 #(File. (str base "out/" (format "%07d.png" %)))
                 (range (count main-view)))]
       (fn [[ main-view proprioception target]]
         (println target)
         (sh "convert"
             proprioception "-geometry" "+20+20" "-composite"
       (fn [& args] (map #(.getCanonicalPath %) args)))
       main-view proprioception targets))))
cd ~/proj/cortex/render/proprio
ffmpeg -r 60 -i out/%07d.png -b:v 9000k -c:v libtheora \

7 Headers

(ns cortex.proprioception
  "Simulate the sense of proprioception (ability to detect the
  relative positions of body parts with respect to other body parts)
  in jMonkeyEngine3. Reads specially prepared blender files to
  automatically generate proprioceptive senses."
  (:use (cortex world util sense body))
  (:import com.jme3.scene.Node)
  (:import java.awt.image.BufferedImage)
  (:import (com.jme3.math Vector3f Quaternion)))
(ns cortex.test.proprioception
  (:import (com.aurellem.capture Capture RatchetTimer IsoTimer))
  (:use (cortex util world proprioception body))
  (:import com.jme3.bullet.control.RigidBodyControl)
  (:import com.jme3.bullet.collision.PhysicsCollisionObject)
  (:import (com.jme3.math Vector3f Quaternion ColorRGBA)))

8 Source Listing

9 Next

Next time, I'll give the Worm the power to move on its own.

Author: Robert McIntyre

Created: 2015-04-19 Sun 07:04

Emacs 24.4.1 (Org mode 8.3beta)