Sunday, December 14, 2008

Getting emacs working

Looked through the clojure-mode source for some good stuff:

unfortunately, clojure-load-file doesn't seem to work due to:
comint-get-source, and then after I start slime *inferior-lisp-proc*.

slime-load-file worked just fine. I wonder if there is a slime-load-all-out-of-date-files. That would be damn useful.

Some shizzle you should know

OK, so something I use a lot in other editors is bookmarking facilities.

So, in emacs we have two functions that I really care about: bookmark-set and bookmark-jump.

So, the idea is to set a point in a given file you know you will need later. You name these so they are unique.

A bookmark is a fully named entity, so these commands ask you for a name.

set: C-x r m

jump: C-x r b

The next thing I use all the time are registers if I have access to them. Basically, I want to select a bit of text and shove it into some named variable I will reuse in different contexts. Registers in emacs are single-character named entities you can put text (and other things) into and get it out of.

These commands will ask for a single character register name to place information into or get it from.

(select region then..)
copy: C-x r s
paste: C-x r i

In general, try to get by without using copy/paste. It *really* is the work of the devil. In any form.

OK, down to the basics. Remember this is aquamacs...
aquamacs-isearch-forward A-f
aquamacs-repeat-isearch , A-g
aquamacs-repeat-isearch-backward , A-G
isearch-backward C-r
isearch-forward , C-s


Don't forget basic navigation, either:
C-p
C-n
C-b
C-f

Finally, I put these two lines in my .emacs file.

(global-set-key [f4] 'slime-compile-and-load-file)
(global-set-key [f3] 'find-name-dired) //This one is *super* useful in larger projects

Trying out the various slime functions, they don't appear to work very well. The awesome ones are 'slime-who-calls and stuff like that; but anyway...

Well, we have hit the point of somewhat diminishing returns. Thus in the next post we will look at the next steps of a non-trivial clojure project.

Saturday, December 13, 2008

Clojure App dev, step 1 -> repl-QT-repl-QT

The goal of this step is to ensure you can start a qt frame from the repl, switch to qt's event handling (thus leaving the repl) but get back to the repl using a button.

This is critically important; I need to be able to run the application and then update it dynamically *without* restarting anything. I can't tell you how many times, while debugging something like mouse picking or moving objects around in 3d space I had to break into the debugger, then check out what is going wrong. Next I would make an attempt to fix, recompile, reload the presentation and then repeat. Over and over and over and fucking over again. And again. What I would like is that I can see a given function isn't working correctly; I just update the function definition *while I am running the editor*. This is goddamn important. With this working complex, really useable features are much easier to get working correctly.

OK, so here we go!

It should look like:

(start from repl)
open qt frame with one button, return to repl.
switch to qt's event handling thus effectively leaving the repl
push a button and return to repl *without* closing frame
add a menu item to the existing frame, thus ensuring we can mutate the datastructure and get a good update
exit main frame; thus going back to repl.
see what the state of the main frame is; we may be able to open it

This relies on one key assumption: that I can exit qt's event handing without closing the window. I believe I can but I don't know.

If this assumption doesn't hold then I can always store, latently, the commands for creating the UI. Then when I want to repl around with shit I can destroy the entire UI, update the commands to add new elements, and re-create. This is heavy-handed and rude but it might work. It will remove perhaps 30-40% of the functionality that want, however so I will really try to avoid this.

Reading the QT documentation, it looks like you call QApplicationCore.exec() to get things going and then you can call QApplicationCore.exit() in order to return from the event loop. So your return-to-repl button should just call exit, perhaps. There is also a processEvents call that you could call. Essentially, you could loop over calling process events and have your button set a global that tells you to stop. But if exit works then that is the sweetest.

Simple clojure QT example

This should get me started. There is something really weird when you are learning a new language. You don't know how to create a hash, or vector, and it seems every character you type is wrong. It takes a couple days before you (or perhaps just I) get things going. On the other hand, this level of comfortability with the language has benefits. You focus far more on algorithm than details; this leads to better code.

For example, if you lived in C for all your life and was asked to do something one of the first things you would visualize or plan out would be the memory access system because the most complex part of most C programs is based around memory handling; at least as much as what the program is meant to do. It takes a few other languages with garbage collection before you begin to think at a higher level and then work down to the memory level if you have to.

Side tracked:

BillC figured this all out first
This too. It hasn't worked for me yet, though

Well, this blows. I have been trying to diagnose an error for quite some time now, it looks like:

[Thrown class java.lang.ExceptionInInitializerError]

0: com.trolltech.qt.QtJambiObject.(QtJambiObject.java:57)
1: java.lang.Class.forName0(Native Method)
2: java.lang.Class.forName(Class.java:169)

Checking shit out, it isn't immediately clear what is going on. Time to research how to debug this error....

The demos run just fine, so I know it is possible to run applications. Trolltech -helpfully- included a binary starter to the demos so I can't easily see what is going on.

Holy shit that hurt!

Why God Why Doesn't It Work?!?1!1?

**Don't use JDK 6**

Nice, 2 hours of fucking around with java madness. I finally figured out the magic google code that would find the problem exactly:

java.lang.UnsatisfiedLinkError libQtCore.4.dylib

So I need to amend my other post.

Some more things are working. It appears that QApplication.exit actually closes all open widgets. This isn't exactly what I want. So I will just call process events from a loop and have a button set a variable to break out of the loop.

Thus I would like to be able to define a variable in the namespace that is something like "process_events_running" and have the single button in my application set it to false. Then I will provide a custom exec function that calls process in a loop checking that variable.

The simples way to do this would be a closure. I don't immediately see how to do this, but I remember rich hickey stating that lambdas are "ICallable" or something like that. I know that the signal system in QTJambi uses reflection, so if I just pass in a closure as the "this" argument to connect, and pass in the function named call on the icallable I may be able to get somewhere. The interface is Callable:

user> (lambda `(println "hello"))
; Evaluation aborted.
user> (fn [] (println "hello"))
#
user> (set x (fn [] (println "hello")))
; Evaluation aborted.
user> (def x (fn [] (println "hello")))
#'user/x
user> (. x call)
hello
nil
user> x
#
user> (x)
hello
nil
user>

Still not finished, but I need some food.

OK, nice microwave pizza and I am back.

So, passing in a closure to connect works fine. I do it like this:

(defn exec []
(def exec_var 1)
(while
(== exec_var 1)
(QApplication/processEvents)))

(defn create_app_frame []
(ensure_app_init)
(let [app (QApplication/instance)
button (new QPushButton "Go Clojure Go")]
(.. button clicked (connect (fn [] (def exec_var 0)) "call()"))
(doto button
(.resize 250 100)
(.setFont (new QFont "Deja Vu Sans" 18 (.. QFont$Weight Bold value)))
(.setWindowTitle "Go Clojure Go")
(.show))
button)) ;return the button for further reference

Now I need things to sleep because I am chewing up CPU by calling process events over and over again. There is a hasPendingEvents call; so now all I need is how to make the system sleep in a platform independent way. Java has Thread.sleep in the language and that is that.

I didn't mess with the datastructures of the frame yet (mainly because I just have a button). But I have a QT app, running from the repl and most importantly returning to the repl at the push of a button. I had to do a lot of work for this first app; get emacs working, run up against problems with the mac java implementation interacting with QT, and learn a little bit of clojure (which was the best part). The benefits are huge, though, because QT is a good platform to move forward on and because clojure is an order of magnitude more powerful than java; and I personally believe that developing from the repl is much more powerful than developing from a compile/run standpoint.


At this point, I would love to upload all files related to this. I can't, so I started a github project where I will put all the code.
Git R Dun

OK, to quickly review what you will need to get shit working:

QT - I had version 4.4.3_01
java - 1.5.0
aquamacs emacs
git,svn,cvs, and the latest versions of:
(svn from various other places)
clojure
clojure-contrib

(from joshu's git repository)
clojure-mode
swank-clojure

(cvs. I wish this project had some better regression testing systems)
slime

Watch every single presentation here:
Rich Hickey

Watch them again until you really get it.

Next up will be an emacs post; I need to remember how to use/navigate within emacs and how to integrate with slime better.

Java application development

When writing a new application in Java you really have several choices. You can use the SWT, swing, and perhaps QT to name a few. On Linux, you can probably use some GTK wrapper but I don't know.

Everyone wants extensible applications, but what does this really mean? The answer to this question is kind of important. There is eclipse-style plugin architecture; plugin with a capital P. You will not write a better plugin architecture than this. You may write one that is more minimal that suites just exactly your needs but it won't be better for sure.

I am stuck at this point. I would in some cases like to build an application off of the eclipse system but there is something about it I don't like; mainly the steep learning curve. My application will not look like eclipse and I am not certain it should use the workspace API. If you don't use the workspace plugin as the basis for your application, then the way I see it most of eclipse isn't really that necessary.

I am not certain I like how the SWT looks, or how responsive (or otherwise) it is. Swing is out of the question because I am going to use openGL. QT is my absolute top pick, but then I have to design the application framework myself.

The question is it worth it to spend serious time learning the eclipse application development foundation *and* figuring out how to get it working in a live coding sense, or should I just mess with QT?

It really bothers me, more than I would like. I have built applications before; the framework (the command architecture, input handling, the UI sub-structure) is really pretty detailed and takes a bit of effort. If you want chording inputs like emacs then you have to do a lot more work but that is honestly the only thing that makes sense. Now you want internationalization? That will take a bunch more work, something else eclipse has already solved well.

On the other hand, if you want great user-interaction; arguably eclipse does not have this solved. I don't like using eclipse; I always get into weird states with it and it really is slow. I would take visual studio over anything in most cases, followed closely by vim or emacs. Eclipse is a distant, distant choice I would only use if it really made sense. Perhaps this is ignorance on my part; but it is never enjoyable for me learning a new, large, non-trivial IDE. Visual studio crashes like it was born to but it is still pretty quick to do things with.

Actually, thinking about it I guess I could try to write some clojure extensions for eclipse before I jump into doing things with QT. The problem is that the UI lib is only 10% of the problem space that I don't care about any more. Everything else I mentioned is also a part of the space I don't want to solve in some amateur, one-off fashion.

But this is the thing. I am not going to use a system that doesn't support live coding. This means either I can launch it from the repl and come back to the repl by pushing a button (thus exiting the application's even handling) *or* I need some repl abilities *in* the app itself.

If I went with QT, I could run everything from the emacs-slime repl. If I go with eclipse, I will need to first build and extension that works with the repl.

Following my heart at this point means QT. Following my head means eclipse.

I guess that answers the question. I will mess with QT and hopefully come up with stuff that could be moved to eclipse should the need arise.

Clojure on mac osx

1. Download java 6
--EDITOR Do not do this if you want qtJambi to work on MacOSX.
--This is because there are only 64 bit java 6 implementations for OS X, and
--Trolltech have not released 64 bit compatible libraries *yet* for OS X. They don't appear
--to be all the close at the moment, either.
--You need to use 1.5.0, which is probably installed by default *if* you are using 10.5.
--EDITOR

go to here and link the current and currentsdk to 1.6.0
/System/Library/Frameworks/JavaVM.framework/Versions/

2. Set up aquamacs with slime:

http://groups.google.com/group/clojure/msg/ecbf7f87343d7f3f

Make sure to get both swank-clojure and clojure-mode from joshu's git repository

Be prepared to update the four of: clojure, swank-clojure, clojure-contrib, and slime so be sure you are get subversion/git/svn versions and can easily go into a directory and get fresh ones.

Yay! I have slime and clojure up and running!

That really wasn't very hard, now I can mess around with clojure a bit and try some things out. It is sweet, open up emacs and press f5 and there you go! I saw some errors flash by at one point but right now I don't care, the repl appears to be working (along with tab-completion of function calls).

Except now I have to remember all the emacs tricks that I have converted to vim tricks and forgotten about...

Why are emacs key bindings so crazy?

Tuesday, December 9, 2008

What is Eclipse?

When you think of Eclipse (I will refer to it as eclipse) you probably think of a large, corporate java editing platform. This is what it has become but it isn't what it is at its heart.

The core piece of eclipse is a plugin framework. This is the extent of my eclipse knowledge.

Given nothing else, eclipse is a system for managing plugins. Plugins can publish interfaces and 'extension points' (whatever that is) and they can state what interfaces they require for correct operation. They do this in an xml file, so Eclipse can delay-load everything based on a large plugin xml database.

The end goal of this project is to enable a live-coding environment in eclipse that is editing the eclipse application it is running in. I will attempt to do this using clojure but I don't actually know how possible this really is. It should be possible and if I am successful it would be a darn cool way to specialize the editing platform.

First off, I want to really understand what eclipse is. What datastructures are involved and how do you talk to them. I will want to understand what eclipse needs for various types of plugins and I will attempt to ascertain how amenable the system is to live-updating while it is running.

All I really need to do is figure out how I can build an eclipse plugin in clojure, however. There is a clojure plugin for eclipse already that is written in java but I will be god damned before I spend my free time working in java. I did that once and it took months of studying pure math for me to undo the dumness. I was awash in waves of dumnity and I almost drownded.

I am just going to study things like I usually do and update this blog as I study.   I am currently downloading the eclipse framework.  I already downloaded clojure and have no idea how to do anything with it but the repl.  Perhaps I can extend eclipse using the repl?  Is there (load-file "blah") functionality?  REPLs are nice, but typing things once is nicer.


Gotta start somewhere....

Opening the eclipse I was downloading. It is missing the "What the fuck are you?" button.

Reading the manual about the workspace, it appears that there is a small runtime kernel and a workspace on top of that. Where is the definition of the runtime component?

One thing going through my mind is 1. how ugly the interface is and 2. how slow it is to update. This isn't ideal; there really shouldn't be noticeable time between moving a window and seeing it change. I have been worried about this before and thus using the eclipse plugin framework with the QT UI framework definitely has come to mind.

Another thing I don't like about eclipse as it stands as it has moved some substantial project management functionality into its UI. These things seem orthogonal concepts to me...

perhaps a little closer:
http://www.eclipse.org/equinox/

Downloaded the osgi specification. I know this is overkill but I gotta. I hate it when you have to register for things to get them btw. It is a fucking spec, it is in everyone's interest for it to become common knowledge...

--"This aspect of the
Framework makes an installed bundle extensible after deployment: new
bundles can be installed for added features or existing bundles can be modi-
fied and updated without requiring the system to be restarted--"

This would seem to be the key enabler for a live-coding environment...

OK, reading a bunch about this stuff is interesting and I guess now it is time to start taking a bunch of other steps.

Overall, the OSGi system is an entire execution environment. Whereas when you compile a basic java program you get access to all of the java libraries you have referenced from your jar file, an OSGi bundle has access to libraries that it has declared it needed through a set of constraints.

So the classloader is specialized depending on your bundle specification and the existence of other bundles in the system. The environment takes care of things like loading two different versions of the same object, unloading and reloading shit and various other details.

What I was thinking was a lot lower level. If I implement stuff in clojure such that any interface I give to eclipse has a pointer to an implementation then I could conceivably just update the implementation pointer when I load the new file. Assuming I can find the interface I handed back to eclipse but I would assume in a forever-running runtime this should be possible. This would work regardless of OSGi nonsense, but at least there is one less term I have to filter out due to lack of information.

Next up I get some basic clojure editing working with my aquamacs emacs system. I got stuck earlier on classpath issues, but I need to read a lot more clojure editing environment tutorials. I am not certain what classpath means on a mac but I couldn't find a classpath environment variable....

Chris

Monday, December 1, 2008

Welcome to the club

I just paid another fee into the club of total pwnage. For those of you who don't live on computers, that means that I just got the receiving end of some shit. I had a part in that shit and it was wonderful but now things are all going lots of different ways.

As a lot of stories go, this one involves two women and one man. Their names and identities are withheld to protected the condemned.

One was an ex-gf, and one was a friend of the ex. There was some sleepin', seducin', and general debauchery goin' on, among things not fit to mention.

Needless to say, it all came crashing down hard with the man a victim of his own desire (first time for everything, hah!), and the women both victims of their own petty rivalries. Mutually assured destruction created mutual destruction.

There is a certain poetry to doing something really really dumb. Among that, ideally a certain amount of pain.

The whole situation just seems pretty contrived, really. I might also mention that the ex spent the night at the new boyfriend's house. And yet she was so angry about things in the morning.

The guy was seduced. A beautiful woman shows up at 12:30 at night at the house and what? You think the guy, who is not dating anyone, is going to just chill? Seduction rocks! Yay!

If anyone wants to try seducing me, btw, please, don't hold back. Furthermore, don't be surprised by the consequences.

Tuesday, November 18, 2008

Abstractions, Symbols, and Intelligence

Buckle up, settle down, and get ready for something exquisite. I am inspired tonight and what I present here should take a while to really sink in.

The primary use case of a computer is to augment human intelligence. We know that much. I feel, however, that in order to build an intelligence augmentation device you really need to understand how intelligence works.

People can only handle a certain amount of information at one time. No matter how smart you are, the amount of simultaneous details you can keep in your head isn't really that high.

You have to build abstractions; and a given abstraction of some concept I will call a symbol.

It is funny how when you are learning new things how we fight building a given symbol. A lot of times, after something becomes clear I am amazed at how concise it turns out in my head. Many times I have really worked hard to figure out why I fought a given concept so hard. Math is usually like this; when we are alert but relaxed we can pay enough attention to really learn it and enjoy the feeling of new concepts bouncing around in our heads.

Given time you get comfortable with a symbol and you start to incorporate it into you view such that you don't even notice it. Then the symbol gets semi-randomly combined with other symbols and if the new symbol is interesting or enjoyable you remember the new symbol. This process continues indefinitely in all of us all the time.

You should take time to learn new concepts that are interesting and different from what you know. As you build layers of abstractions and symbols to really adapt to the new experience you will make associations and new connections between things you have really known for a long time but been unable to see from the new perspective. Thus learning N brand new and different things really teachings N^2 (N-squared) or more new things; intelligence and learning are at least exponential (this relies on the assumption that this really is a very new thing)!

In any case, we all think by making symbols and manipulating them in some sort of systematic fashion.

The symbols you are allowed to make in a programming language, btw, directly relates to how exact and high level of a sophisticated of a concept you can express with it. Higher level programming forces you to make these clearer and higher level abstractions. The type system provides the rules for combining the symbols you make with the programming language.

Now lets talk about programs and get completely specific and practical. Most programs are very limited in the forms of symbols they allow you to make. Here are a few examples of building symbols:

One is an abstraction over a collection of objects. I have all these paragraphs organized in a chapter. Now I can re-arrange these chapters and form a book. That is a literal hierarchy of symbols. You can bet that in order to have a book laid out in a sophisticated manner the author has built a lot of abstractions about what the chapters mean and exactly how the flow of the story wanders through the literature.

Another example would be that I have these groups of formulas in a single excel spreadsheet. In the user's mind, they are going to build some symbol out of that spreadsheet that allows them to reason about its capabilities without knowing every single line of code in it.

Second is the ability to extend a given symbol with new information. This form of abstraction is akin to a master-instance relationship where you are in effect saying "this item is just like that one, but it differs in a few aspects here". Artists really love to use this form to create crazy interesting concepts. Dj's, Hollywood, video game makers all use this form of abstraction to some greater but mostly lesser benefit.

It is very common on the web where you have a templating system so that a given site will have web pages with some standard styles, heading and footing but override a lot of other things. Most systems use templates incorrectly, however. One they aren't supposed to be something you copy and start from. If you can't change the final product by changing the template then you haven't built an abstraction; you have just added more information. In the common case of Word documents, if I have a bunch of documents based on a template I should be able to change the headers in all the documents by changing the header style of the template. This doesn't happen (you might argue that it is safer and that is fine; I am talking about being able to build complex abstractions; this requires re-evaluating concepts that you might already know and thus presents the risk inherent in that you are changing existing symbols you are talking about ensuring that you *can't* damage existing goods which means that real learning or abstracting isn't happening).

You see this in programs a lot where you like a set of programs to have a standard look at feel. You get used to using these programs quickly because you aren't focusing on building new symbols from low level concepts but instead you are understanding the differences between what you are doing and what the other program was doing. Ctrl-C in windows is copy in most programs. This means that while its meaning may be context specific the general mental symbol that stands for copy is consistent across programs. You needed fewer symbols to use the new system.

Lets look at how someone learns a song. You get the main feel of the piece; this is the first symbol you build. Then given this rough, broad symbol you continue adding detail and forming new symbols until you build the song out to a tolerable richness or accuracy of reproduction.

I can't immediately think of a third way of learning anything that doesn't fall into a combination of the two ways I stated before.

This is where orthogonality of concepts really comes in. Clear, orthogonal symbols are composable in ways that rough, non-orthogonal concepts are not. This sucks because creating new symbols, either through grouping or through templating is what learning really is.

Earlier tonight I realized that my picking technique was too strict. Playing guitar, doing some random drill I found out of a book to warm up my hands and clear my mind I realized that I could pick this drill several ways; I didn't need to really develop some strict strategy and pick it the exact same every time. I then tried many techniques to see which ones felt the best; the drill had a lot of jumping from one string to another and to get it to work well I tried a hybrid-alternate picking style where I arpeggiated the jumps. This meant that if the jump was down several strings I used a down-stroke on both strings, and then used alternate up-down picking where I was playing a line of notes on a single string. If the jump was up several strings then I used an up-stroke for both the last string before the jump and the first string after.

Then I realized that I could also pick it with a strict alternate picking style where I could pick down on the top string, thus moving the pick towards a lower string *move completely over the lower string* and then pick up on the lower string. This meant that I was technically doing more than the most minimal amount of work in order to pick the drill but I realized that it allowed my wrist to make a more natural and better timed stroke and I decided since I had the control to do the strict style I liked it better because it felt like the timing was easier.

This was an example where my symbol I had built for picking -- it had to be the most minimal cost route to getting things done -- was way too strict. The truth of the matter is that wrists are damn quick and for most situations easily move the pick fast enough. Doing things in the most comfortable way that allows you to play the notes is more important than mechanical efficiency. Knowing this then gave me a lot more confidence with songs because I started noticing all the places where I really had somewhat inconsistent picking but because I have worked on control so much it doesn't affect my ability to play the song.

This form of abstraction, templatization, becomes to strict when the template has too much information. This means that it eliminates possibilities that really should be valid.

So what is the point of all this? Just for once I am going to ask for some reader-participation. I just gave up a lot to come up with all that above, it has taken me years to figure all this out. So please, if you enjoyed the above explanation lets take it to the next step.

You have two distinct forms of building symbols, aggregation and templatization (or master-instance if you like).

1. Are there other distinct forms of building symbols?
2. How could you apply these ways of building symbols to the programs that we use? An example would be if you had a Word document, are the different ways you can aggregate and templatize the document? We are familiar with document templates, can you have a meta-template where you have a common template across templates? What about aggregations of the templates, what would that imply? Aggregations of pages make a chapter, but under what scenarios do you want to use a page as a template or master of another (revision control comes to mind).