Iv just finished watching stuart-clojure-presentation where the issue of singletons is mentioned (global & thread local ones), a note from the audience mentions delay as a solution, REPL-ing a bit shows how this works:
user=> (defn singleton-factory 
(println "creating object")
(+ 1 2))
user=> (def singleton (delay (singleton-factory)))
user=> (defn usage 
Stuart does mention contrib singleton which has also per-thread-singleton.
user=> (use 'clojure.contrib.singleton)
user=> (def t-singleton (per-thread-singleton singleton-factory))
user=> (defn use-twice  (+ 1 (t-singleton)) (+ 1 (t-singleton)))
user=> (defn use-twice-no-singleton  (+ 1 (singleton-factory)) (+ 1 (singleton-factory)))
user=> (. (Thread. use-twice) start) ; each Clojure Fn implements Runnable
user=> creating object ; REPL prints on the wrong line
user=> (. (Thread. use-twice-no-singleton) start)
user=> creating object ; same thing
This should come useful when working with all those Java mutable objects.