Fully Typed LISP

I have been thinking about this concept for at least couple years now, pretty much every since I stumbled upon LISP in the first place.

Let's first outlay a couple assumptions I have built up over the years.

Compression potential is essentially abstraction potential, or the level of abstraction you can comfortably maintain in a programming language. Assembly has no compression potential aside from functional decomposition while a programming language with a programmably compiler has infinite compression potential. Lisp and Factor, (moreso factor than Lisp for whatever reason) have almost infinite compression potential. C++, using templates and X-macros, has actually decent compression potential. Functional languages, with closures, discriminated unions, and pattern matching has pretty good compression potential for some problems. For others, simple imperative languages clearly dominate.

Optimization potential is the potential for the exact same piece of code to run faster over time due to advances in compiler (or runtime) design. Lisp again had seemingly infinite optimization potential but we later figured out that much of this potential was impossible to get to (no compiler is sufficiently smart) although you will notice that SBCL is often 3rd in the language shootout game meaning if you use it in certain ways, it can be quite fast. A completely abstract programming language like a true functional language such as Haskell or perhaps prolog should also have very high optimization potential because you are programming to a completely abstract environment.

Correctness potential is the potential for the compiler to check that the program is correct. This requires something of a type system and all compilers, dynamic or otherwise have it. Again fully typed functional languages such as Haskell definitely have this exploited while in C++ the exploitation is up to the discretion of the programmer and in LISP you have to hope that your tests cover all of your runtime situations because in dynamically typed languages the compiler isn't helping you much.

One key difference between the type of problems caught by a static type system and the type of problems caught by some sort of runtime testing, be it unit tests, integration tests, system tests, black box, white box, or however you want to categorize the tests. A type system guarantees that certain types of problems won't occur. But it doesn't guarantee that the program will ever produce the right answer for any input, or that it will run to completion (in most causes, I know Haskell catches some of these). Thus you always need unit tests, but unit tests can't get complete coverage of an even moderately complex system.

In a highly complex system, such as a physics engine, unit tests help but can't hope to cover even a small percentage of the range of test cases. Especially when the right answer is highly dependent upon human perception of what looks correct or some other fuzzy definitely. So a type system is absolutely necessary.

So taking all of this into account, I want to outlay the vision and some syntax for a new language, a sort of fully typed lisp-syntax language.

Lets start with the preprocessing stage. At this stage we can't have any types, any sort of compiler extension is going to be roughly of the type of text->text. Traditional LISP macros work in this range. The output of the preprocessor goes to the typing stage where the static typing system starts interpreting the data and providing feedback.


So, in this stage, anything that looks even mildly like lisp gets passed through. We have at the moment only one mode of interaction with the compiler, def-parse-fn.

The compiler language at this point is traditional, untyped list with a macro syntax of clojure. Unlike traditional lisps lists evaluate to themselves unless their first member is something declared through def-parse-fn. Inside parse-fns, we have the normal family of escape characters of a lisp, i.e. backtick, tilde-evalutation, etc.

Now we get to another stage, where the type system can start to function. To do this we must, unfortunately, define more syntax.

(defvar <vartype>varname (init-body)(attribute-list))

vartype may be inferred from the initialization body. But it can also be specified formally to ensure the initialization body matches what one expects.

(defn <rettype>fnname (<type1>arg1 <type2>arg2)
(body-list) (function-attribute-list))

Like vartype, rettype can be inferred from the function body but it can also be specified formally. Also both types have matching dec{var,fn} members for simply declaring these objects to the type system but and expecting them to be found elsewhere.

I can't see how clearly this shows through in the blog but there can't get a space between these items, and types are always inclosed in <type-list>. This makes it a bit more familiar to people used to c++, java, or C#. It also allows them to pass through the first stage unchanged and with no chance of interpretation.

So, lets talk about defining a structure, or a named data layout possibly associated with some functions.

(deftype typename
(struct
(field name (no-arg-initializer body) (field-attribute-list))
(field name (mutable))
(constructor (arg1 arg2 )
(init-field fieldname init-list))
(destructor)
(property (get-fn) (set-fn)))

I could go on with member functions, but you get the idea. In this case, the typename names a defined layout, in memory, along with functions for creating them and destroying them. Fields can specify default initialization values for their members and such, but structs should also be able to specify that they a much dumber, meaning they can be treated like PODS (plain-old-datatype members)

Given a type, we would like to associate functions with the type and with an instance of the type (static vs normal functions).

(assoc-type-fn typename fnname) function is associated with the type.
(assoc-fn typename fnname) function is associated with an instance. One of the arguments to the function must be of the instance type and this argument will be expected to be elided when calling the function through the instance.

These functions may be used outside of the type definition to associate functions with the type. Whether the association is done inside or outside of the type, if you are calling a function via an association it looks like this:

(type-name fnname arg1 arg2)
(instance-var fnname arg1 arg2)

Breaking the normal lisp tradition allows intellisense to work normally. So in essence, instance are functions, where the first argument is the function name telling you which function to call. Ditto for types.

An important point is that getting/setting a field is no different syntactically than getting/setting a property.

(instance-var set prop-or-field-name value)
(instance-var get prop-or-field-name)

So I give a natural migration path that does require a recompilation but doesn't require much else.

At this point we have at least two other interesting meta programming possibilities. The first would be def-fn-macro.


(def-fn-macro macro-name( arg1 arg2 ) (body))

Note that these

Another macro type would be a type macro, where a the compiler expects a new type defined, def-type-macro.

The final statement of a type macro *must* or return a new type. This type may have functions associated with it, even static functions but it must be a type.

Now we turn our attention to the type system.

The type system for a language like C++, removing auto-casting, is very simple. It simply asks if the types match for assignment and function calls.

The type system for Haskell is a theorem proving machine.

I would like to start with the C++ type system, and carefully and transparently work toward the Haskell type system in a way that doesn't interfere with existing code. I would also like the type system to be compile-time extensible such that you can have a type system macro but I don't have a clear idea what that would look like yet.

One difference I have noticed between what I want to do and C++ is that in C++ changing a type, which is a compile time object, requires a runtime function call (which of course may be elided or inlined, but whatever).

Let's say we have a function that only works on integers within a certain range. I would like to declare that function with those parameters declared in the type of the function call.

But I don't want changing an arbitrary integer into the correctly typed integer (or back) to mean any real copy operations, even with the naive compiler. The type is something that, while it may require a runtime check, doesn't fundamentally change the underlying data in the system. So it shouldn't require an data copy or transfer operations. The compiler should ensure this runtime check is done before the function call. You can do this type of programming in C++ but it is tedious and perhaps that is the best you can do.

But one large difference between the way C++ is used and Haskell is that in Haskell people really do try to use the type system to ensure program correctness. This type of programming needs to be encouraged.

Also, Eiffel's contracts need to be encouraged but I believe the way to do this is to enable them to be done in the type system. Then let the compiler work out exactly when and how to check the pre-condition and post-conditions and such. Code that doesn't maintain the types correctly doesn't check the pre and post conditions.

Now, moving quickly, the runtime. Unfortunately, the standard lisp runtime is tough to do if you are type-checking and compiling things although haskell seems to have done something like it.

The LISP runtime, however, allows you to replace an existing function *in a running program*. This is actually quite key when debugging certain types of problems, and can make almost all debugging much faster. It also eliminates the possibility of inlining and a whole host of optimizations so it needs to be used sparsely; or at the very least the programming needs to be able to enable/disable it for a set of functions.

In addition there is the technical problem of calling a specific function from a generic data system (the repl is of course, generic). This means that every function that needs to be called generically at least needs to have a generic thunk wrapped around it that takes the arguments from an appropriately sized pointer array and writes the result back to an appropriately sized buffer.

Anyway, I am attempting to write this typed list language. Note that I didn't say anything about a garbage collection system. I think that while those do make programming easier, they are a large, unpredictable solution to a set of smaller problems. Regions, memory pools, and stack based construction/destruction all contribute to lessen the need for any sort of garbage collection.

Ditto for multithreading. I think clojure's current facilities for multithreading are exactly the correct ones (especially its stm implementation) but I also think some of the latest advances in the C# compiler, basically enabling continuations, are also very important.

Finally I am not talking about locally defined closures yet. The three main things I think make programming much more concise aside from a programmable compiler are closures, discriminated unions and more generally pattern matching. Out of those, only closures so far need extensive compiler support and I will get to that support when the time comes.

The other feature of C++ that absolutely must make it is being able to attach a set of statements that have to be done at any exit of an enclosed block of code. This is a generalization of using objects on the stack whose destructors do cleanup, thus ensuring that code gets run no matter how the function is existed. The generalization would look like:

(on-exit (stment-list)) and the compiler would ensure that the various details around this are done.

Exceptions, at least c++ style exceptions aren't mentioned because they are non-trivial to implement and are actually still controversial. I do feel that their use in certain contexts simplifies the code but I also know from experience that you can live happily without them and their overuse makes the code much less predictable. LLVM provides pretty good support for them, however, and i do intend at some point in the future to support them.

I am working on an initial implementation of the preprocessor, it has taken me a month alone to go through the various details to solidify my thoughts on the language compiler this far.

The overall vision is that a programmable compiler will beat a sophisticated one both in the short and long runs, at least in terms of programmer productivity. Furthermore, if we carefully break down the most useful features of most languages we can add them slowly and carefully when necessary but they don't need to be boiled into the base compiler. Extending the compiler allows us to add most features in a more modular and optional way.

Can this be faster than C/C++? I don't know, gcc has 30 years of optimizations behind it.
Can I write safer code in this than in C/C++? I don't know, that depends on how sophisticated we want to make the type system.

I am betting than a simpler low level system with good programming and modification possibilities beats more complex and developed systems. Over time I know that this means that community agreement and understanding of shared abstractions is much more important than with a language with a fixed, standardized definition. A good digression is why this doesn't work with c++. Why does everyone roll their own vector class?

This does give a good platform for experimentation for higher (and lower) level language features.

I am a dancer

This is extremely difficult for me to say.

Something has changed in the last year or year and one half and I am really no longer a computer scientist. I am quite good at it and will always work and study hard where it is concerned. But it isn't my passion any longer.

I code when I am at work. I don't code on the weekends and I don't code at night when I am done at the office.

Every evening I am either dancing or stretching and watching videos of dancing.

I do ballet. Twice a week. I bounced around the room attempting tour jetes today and one of my knees is sore to prove it.

I just spent an hour watching anaheim ballet videos.

Anyway, I code for a living. But I am a dancer.

Chris

Lisp C preprocessor

In my daily job I program in C++. It has been that way for 10 years and it looks like it will be that way for 10 more years.

I understand c++ quite well but there are aspects of the language I find quite tedious. Actually, look, I don't really care to justify my design decisions. I am going to attempt to write a lisp-like language that generates C.

Hopefully I can make it enough like C that the parenthesis-to-bracket compilation is cheap and fast. I would also like to take some stabs at what I believe are C's largest problems but I need help because this is really tough and I want to incorporate more ideas from other people.

So, to cut to the chase:

The point first and foremost is to allow writing C code at a higher abstraction level. I want to write:

(let [sum (reduce + 0 (int-array 1 2 3 4))]
(cout sum))


I want this to compile down to:


{
int __temp[] = { 1 2 3 4 };
int sum = 0;
for ( int __idx = 0; __idx < 4; ++__idx ) sum += temp[__idx];
cout << sum;
}


and given:


(let [sum (reduce + 0 (std-vector int 1 2 3 4))]
(cout sum))


I want to see:


{
std::vector<int> vec( 4 );
vec.push_back( 1 );
vec.push_back( 2 );
vec.push_back( 3 );
vec.push_back( 4 );
int sum = 0;
for( std::vector<int>::const_iterator __temp1 = vec.begin(),
std::vector<int>::const_iterator __temp2 = vec.end();
temp1 != temp2;
++temp1 ) sum += *__temp1;
cout << sum;
}


Furthermore I want to be able to define a reduce like function. Or a map function (with the result provided as an argument; !no malloc assumptions!). You can't do this in either c++ or in c for several reasons. The first and foremost is that in some senses you can but defining closures is incredibly painful and tedious. Or you can use boost::bind and watch your compile times go through the roof (why would you need to compile + test frequently anyway?).

So, anyway, this is my theory. The language doesn't really matter. Look at what people have done with Factor. Basically, *you* are the compiler with Factor; it is essentially a really sophisticated assembly language. As is Lisp, as is C, as are most languages that don't have true abstract datatypes or some of the other abstractions that functional languages have.

The ease of abstractions and the level of abstractions you can created are what matter.

I need to be able to abstract over types like the templating system of c++ kind of allows you to do. I need to be able to write a compiler-level foreach definition that can be extended for new types quickly. And then reduce simply works using the previous foreach definition.

Imagine c++ template language didn't suck and actually allowed you to do anything with the compiler you wanted to. C++ programs would become a *lot* more terse and probably much more correct. Its like being able to define you own dsl *very* quickly and efficiently and have it output to c++ code.

Then imagine a system that was designed from the ground up to make creating shared libs on any platform and using them from other languages really really easy.

Now imagine some CUDA and OpenCL bindings so you can do hardcore parallel computing very efficiently. Mated with a language like clojure you would be able to take advantage of several different levels of parallelism (machine,CPU,GPU) all very efficiently.

Success Intelligence

I am reading a new book and it asks some interesting questions. Here is one interesting example (I decided to do this while meditating):

Ask yourself "What do I want?"

Think about it for ten minutes.

Ask yourself "What do I really want?"

Think about it for ten minutes.

"What do I really really want?"

Think about it for ten minutes.

There! You have had your mediation time, a little bit of self exploration, and some life advice all in the same 30 minutes!

Chris

Multithreaded UI

Lets start this discussion with this assumptions:

It is desirable to have your user interface as threaded as possible.

Thus we guarantee less about the latency of the updates and more about the potential to display extremely intricate user interfaces on low-parallel machines.

First off, lets talk about levels of parallelism. First off, we are counting threads of execution (TOE), not cores:
low : On the order of 10's of cores (1 - 99 TOE)
mid : On the order of 1000s of cores (100 - 9999 TOE)
high : 10,000 TOE and up.

Currently, our machines exhibit low levels of hardware parallelism. Some big servers exhibit mid levels of hardware parallelism, and graphics cards as well as supercomputers exhibit high levels of hardware parallelism.

I want to think about extending UI implementations to low levels of parallelism from a single, or largely single execution model present today.

As a simple model, lets take a 3d view with an object along with a palette where the object's position is displayed. We have two different types of updates, where the user drags the mouse and when the user enters a discrete value.

Generically, we have some data model. We have two different views of the data model with most likely extremely different datastructures used to display the data.

We furthermore assume that rendering is only safe from one thread. This thread renders both views according to some schedule, either on dirty or as needed due to a framerate requirement. Breaking this assumption usually has extreme performance implications *or* it is not allowed (i.e. hard crash), at least with current rendering API's (opengl, .net, etc).

Starting from concrete and moving to generic, here is a design that has been slowly manifesting itself in my mind.

Per view, there is some translation from the model to some very view specific datastructures where the view's particular renderer iterates over them and takes over.

I propose that there is a thread queue where each view places its individual controller. So the entry points to these translation steps can all be started from parallel. The product of these translation steps is thrown into a threaded queue where the render thread takes over and iterates over them telling each view to update on its own accord.

In any case, something changes the base model. Now all the controllers need to have a chance to look at the model and decide if they need to produce a new render datastructure. This translation step can be done in parallel for each view. Furthermore, this translation step should translate the model quite far into view specific structures so that the render thread finishes rendering as fast as possible.

So, functionally, we have:

mutator->model, model->view structure(s), view structure(s)->renderer.

Ideally this design would allow for views that have simple translations to finish and render quicker. Thus a view with a particularly involved translation step to render wouldn't have the ability to throttle the application.

So, we can at least speed up rendering of complex applications with multiple views in a very basic sense without allowing an involved view to slow down the application. Each translation step should be further threaded if it makes sense for the amount of data and the problem domain but we haven't said anything about that yet so we can't assume anything about the specific translations.

UI's are not written with this design but if you want truly sophisticated graphics effects it will require allowing the machine more room to process complex views and components. Thus rendering the entire application cannot be held up by the rendering of a complex view and user interaction should not be noticeably slowed down by the rendering of a complex view.

Ideally you can also design a system where the more expensive views receive proportionally more compute time in a multicore system and this type of application design requires thinking very hard about your translation pipeline from model to view.

Chris

App design (internal, read if you want a headache)

I am building an application to create beautiful interactive graphics. This isn't a game engine; it will be used to create really advanced applications.

In any case, I need to write out some of my design ideas because they are kind of stuck in my head.

Application programming with functional datastructures is different than with imperative or mutable datastructures.

This is because with mutable datastructures, you need to take into account the changes to the basic model such that you can inform the rest of the system about what is going on.

Since functional datastructures are immutable, the rest of the system can use a simple identical? comparison test. If not identical to what it was last time (which is a pointer comparison) then something changed.

Now you can segregate your data into groups, link the disparate pieces through ids and you should end up with a system that has amazing undo capabilities *and* doesn't take a lot of though to program.

Undo is simple and efficient. Just save the old datastructure. Since you know that the datastructures will have large portions of structural sharing, you can bet that your memory usage will grow slowly and efficiently, perhaps more so since with imperative datastructures you have to remember what changed and save this piece of information separately. Redo is similarly easy.

So, basically I intend to have each view simply check the datastructures it cares about and update its internal pieces *if* they are different from what it expects. No events (other than render or update events), no complex transactional-based undo/redo system. Just save the datastructures to named variables and go with that.

It *really* changes the MVC pattern. The controller doesn't need to do nearly as much; it is really much simpler. The views can control how much caching they need by how many references they keep into the main application datastructures. The model can be manipulated very simply and you can still guarantee a lot of details.

Also, using the basic clojure datastructures means I get serialization for free. They serialize pretty naturally.

So enough about that, I need to figure out how I am communicate things to each system.

Looking at the renderer, it simply needs a set of render commands. These commands a very basic, like render object with these shader parameters to this FBO.

Mouse picking is easy, just run the appropriate render commands in selection mode.

The render commands do not need to be generated in the rendering thread; a lot of pre-processing can go on in other threads to generate very specific and fast render commands.

There needs to be some sort of central state that gets updated and somewhat synchronously processed. So a mouse click should generate a request for a hit test. Next time something renders, that request should be processed and the results, should there be any, sent back to the system.

That hit request should translate into some state changes, like an object being outlined or something. These changes can be done in another thread and a new render-command list be generated that does something else.

So, coming up with very specific design guidelines, I guess this is it.

You are going to have a set of N views. When a piece of the model changes, something should analyze this information and update <= N view specific datastructures. These updates can be done in parallel, so there is some parallelization ability there.

These shouldn't happen on the view or render thread, you should do a bunch of pre-processing on other thread such that you build very view specific datastructures that efficiently translate into pixels on the screen. Each view may be able to further break down its updating system but I doubt they could really do this efficiently. It would probably be better to create sub-view-sections and process them independent of each other. Also, they shouldn't update anything in the event that nothing changed that they care about.

The renderer's view specific datastructure is the list of render commands it is rendering.

So lets walk through an entire event sequence.

You have an object on the screen.

mouse down is sent up to the system from the glview.
a hit test is scheduled for the next time something renders and added to the gl todo list.
lets say you hit the object. This is sent back to the system where processing takes place that changes an application datastructure such that an object is selected now that was not.
This selection event is processed and all the views-controllers are told a new model is available. These controllers, running in the systems' CPU-bound thread, do however much pre-processing they can and change some view datastructures. After the changes are done, each control is told to update.

Now you start dragging. This means that the object should move under the mouse (scale,translate,or rotate depending). The view's drag handler should send the drag event to the system where it will do the math required to move the object around. This thread switch may introduce too much latency but we will see. Next the system will update the scene, views that care should update their internal representations (like the render commands). And repeat.

The question is is there too much latency by jumping from the view thread to updating the application's datastructures? Should that happen synchronously while the view updates happen on the separate threads?

Anyway, needed to get some thoughts out of my head.

real hard fbo issue

First off, lets state some the relevant keywords such that a google search has a prayer of finding this...

GL_FRAMEBUFFER_UNSUPPORTED_EXT
glFramebufferTexture2DEXT

OK, so here is the lowdown. It is critically important for most of the really interesting things I want to do to be able to allocate framebuffer objects that render their information to textures.

On the mac, I would create a texture and set it on the framebuffer object and it would work.
On windows, I would always get the above error. Now, pray tell, how would it be possible that textured framebuffers are supported on the mac but not on windows *on the same machine*?

Well, I checked a lot of things. I thought that perhaps this was because one was using pbuffers and the other was using swing's fbo pipleline. I changed the window to be a awt canvas (thus rendering to the native window surface) instead of a swing GLJPanel.

I set up a simple test case using an example from the web (that failed, btw.)

I downloaded lwjgl, setup a test case, and ran an example (that also failed, btw).

I spent hours searching the internet. This is all in my spare time after work, so I get at most 2 hours a day to work on this stuff and that is only a couple times/week. One serious gl problem can set me back several weeks.

I also printed out every single variable I could think of that might affect this operation (pixel store state, pixel transfer state, read/draw buffer status, etc.)

Anyway, to stop holding people in suspense, what was causing the allocation to fail?

It was because I wasn't setting the min and mag filters on the texture before I allocated the FBO.

Min and mag filter specify what the texture lookups should be doing when there are more textels than pixels (minication) and when there are more pixels that textels (magnifying the texture). They are attributes that affect how the texture is *read* from memory, not how it is written.

In any case, in the document for FBO's it explicitly states that mipmapped images are not supported even thought the fbo texture renderbuffer call takes a level.

If you look at the defaults for minfilter and magfilter, they are GL_NEAREST_MIPMAP_LINEAR.

Well, NVIDIA cards are picky about this. Apparently, just setting the min and mag filters makes the texture object a mipmap texture object and the fbo allocate will fail with unsupported (which is true in a pedantic, brittle, poorly thought out sense I guess).

So anyway, a couple evenings of debugging and trying out anything I could think of and the answer is simple. When you want to use a texture on an FBO, set its min and mag filters to either LINEAR, CLAMP, or CLAMP_TO_EDGE. Or probably any constant that doesn't explicitly say mipmap in it...

Jeez what a PITA.

Getting what you deserve

Money has always been relatively easy for me to come across. I think being half mathematician makes it easier for some to obtain and manage money. In any case, ever since I could make it I have never wanted for money. This is a fairly messed up fact in the grand scheme of things.

Lets talk about what people really deserve. I do not think there is such a thing as deserve. There cannot be for if this were so, humans deserve a lot of pain. This seems extreme.

Millions of children die of diseases that even given my quite opulent life by global standards I would never want to deal with. How could they deserve what they got and I deserve what I get? It isn't like I save children from burning orphanages while caring for the sick in my spare time.

Money, of course, isn't everything but health and money together are a lot. I have both and millions (or billions) of people will never have a chance at either.

So, knowing all of this, how can you be OK with money? Why not run around and devote your life to charity (this also doesn't feel right to me)? Then you surely must deserve what you get.

Except when I volunteered a lot I didn't really feel any better about myself. Every once in a while you would connect with someone and then you would feel great for a little bit, but it never lasted. The organizations weren't very professional and I always felt that a bit more could have been done if someone really took getting the job done a little more seriously. Perhaps my standards of what is professional are a little off.

So in any case, it seems impossible to make a logical argument for deserve. A very large part of what happens to someone really is just dice. Luck, destiny, call it what you will shit happens and sometimes you win and sometimes you lose.

Now there is a really high chance that some of the people that I am arrogantly feeling sorry for for not having lots of scrilla really are a lot happier than I am. Being happier than me really isn't that hard. But I don't feel unreasonably unhappy; I feel like I am happy enough to see opportunities when they lie but I have enough reserve that when something takes real discipline I can stick with it.

So is happiness the point of human existence? If deserve doesn't exist, then what does happiness mean? It would have to mean that you are just OK with your circumstances. You don't really owe anyone anything and they don't owe you.

Its days like these

That fucking rocked. Although it didn't rock at the time, now I feel pretty good.

I decided to get my project working in windows. Since it is a clojure opengl swing program I figured this wouldn't be anything but a quick check and perhaps a couple more function calls.

Thus, instead of quickly starting work I messed around with emacs. My emacs is half-pimped at this moment. I have line numbers, tabs at the top of each frame (in windows and mac), and the theme I like. I have a github project and the same .emacs file runs on both windows and mac.

I also pimped out my clojure setup a bit. I have everything (slime, clojure, clojure-contrib swank-clojure,clojure-mode) all synced up under one dev directory. I can go into each sub directory and sync them, rebuild jars where necessary and just open emacs again and I am running with the latest. No hassle, everything just works (for now).

This all happened yesterday for the most part. I woke up this morning thinking about emacs for some reason and immediately started coding. My project now has built-in slime (remote debugging) support. If you want to connect to the running project, there is a menu item to open up a swank port and start an internal swank server. Now in emacs you can connect to this, start the slime repl, and you are doing pretty good. I don't have to start my program from the repl any more in order to change or examine it.

All this before starting to port my stuff to windows. I did the internal swank server in the mac.

Now, I start running my program under windows and it doesn't work for shit. The first problem that I hadn't anticipated was that for some reason, under windows swing, I am unable to allocate an FBO with a texture. In order to render an anti-aliased scene I have to allocate a multi-sample fbo, render to that, blit to a single sample fbo and then render the result to the main screen. The easiest and by far fastest way to do this is to setup a texture as the downsample-fbo's color render buffer. This works like magic on the mac; no problem.

So, I can't allocate this type of FBO on windows. This sort of fundamental difference on the literally the same exact computer should have fired off a warning in my head. Anyway, I worked around it (using glCopyTexImage2D to copy the read buffer off the downsample fbo. This is much slower but it works) and I saw nothing. I now have two rendering paths, one if you can allocate a texture-fbo, and one for if you can't.

Back to the annoying days of holy shit, what the hell happened?

To make a very long story short, I have all of my java defines enabled for uber fast rendering. Crazy things like:

-Dsun.java2d.opengl.fbobject=true \
 -Dsun.java2d.opengl=true\
 -Dsun.java2d.translaccel=true\
 

Note the opengl and fbooject flags. They change, very deeply, the semantics of the system w/r/t opengl and they speed up rendering a *lot*.

So, in no particular order, here are the things I figured out.

Under the mac, setting the fbo to fbo 0 effectively sets you to render to the main screen. This follows as the mac isn't using an intermediate fbo to render to; it is using a pbuffer which is a different entity altogether although they do similar things. On windows, with the fbo option enabled, fbo 0 is *not* what you are rendering to as the main buffer. Thus for the longest time I saw absolutely nothing. White, in fact, which is a little weirder than black or uninitialized memory in my opinion. I have no idea how it occurred to me this was happening but it was just one of many hard problems.

Under the mac, the gl viewport is set to 0, 0, width, height. On windows, it was more like 0, 200, width height because swing was rending to an fbo and my GLJPanel was only rendering to a portion of said FBO. Thus the viewport was setup differently *if* you have sibling swing controls.

For identical reasons, windows swing implementation is using glScissor tests to ensure you *can't* render outside a given area regardless of messing with the viewport. This is a good idea as you can't trust what another client or control will do. I hadn't taken this into account, of course, because the mac doesn't do it. The effect was that when I rendered to the multi sample fbo I only saw a portion of my scene; it took a goddamn long time to figure out whether my full-screen-quad rendering was messed up or to ensure it was fine and start looking for other options. The solution was to disable the scissor test while I was rendering to the temporary fbos and enable after, before I rendered from downsample to the main buffer.

The weirdest bug *by far* was the difference in the glsl interface. The mac gives you every single shader attribute in order, whether or not you can mess with it. So you will see program attributes that look like "gl_Vertex" even though you can't set them yourself (gl takes care of that).

Windows, on the other hand, just showed me the editable attributes. Thus I had been using the attribute index as its gl handle on the mac but on windows this failed in the oddest ways. It took a very long time to start looking around the glsl code and replace the one implicit reference to attribute index with a call to glGetAttribLocation.

Now the program started crashing when I switched from my functional graphics demo to my wave demo. I eventually figured out this was because I wasn't protecting the underlying gl implementation from my changes. Specifically I was using the gl vertex vbo only for my wave program. My multisample system used gl attributes for everything but I like to explore all options so I did it differently in the wave section.

Somehow, leaving the vertex attribute active made the outer program crash hard. Actually, it makes a lot of sense why I guess. They must be using glDrawArrays or DrawElements to render the swing pieces to the fbo. Anyway, a couple glPushAttrib and glPushClientAttrib (along with their popping siblings) solved that problem quickly once the scent had been found.

Now my program works, both demos, on windows and the mac. This probably indicates it work work on linux, as both windows and linux use sun's swing framework. The mac uses apple's swing framework and thus there are going to be hardcore, non-trivial differences between them.

I really didn't want to try to support windows. But unlike linux it is installed on my mac laptop and I can honestly say that my emacs-foo, my swing and opengl foos are a level higher because I took the time to work through the issues. My slime-swank-foo is becoming somewhat formidable.

Chris

Interesting clojure issue

Worn out from complaining too aggressively

There is a lot I draw from this.

First off, people aren't replying to him on the email list. I find this incredibly lame because I believe, regardless of how he is raising it, that he raises a legitimate point that will cause consternation among people.

The basic thing is that:

user> (= (list 2 3) [2 3])
true
user> (hash (list 2 3))
-1919631535
user> (hash [2 3])
1026
user>

'=' semantics do not match hash semantics exactly. This means that if you intermix lists and vectors as keys in a hash map then you are going to get really odd results:

user> { [2 3] 4 (list 2 3) 5 }
{[2 3] 4, (2 3) 5}
user> (filter (= [2 3] %) (keys *1))
; Evaluation aborted.
user> (filter #(= [2 3] %) (keys *1))
([2 3] (2 3))
user>

You might ask how one would get into this situation, but you will get into it in some subtle ways. For instance you would (map identity [2 3]) and the result would be equal to [2 3] but it wouldn't be the same key in a hash-map.

I personally, now that I know about this, have absolutely no problem working around it. People who are used to c++ have had to work around much, *much* worse (like today I worked around a heap corruption problem that caused a failure in an unrelated piece of code).

People who are used to Java are used to a *completely* normalized environment. This is the ideal that we all strive for; no idiomatic problems that you have to deal with; just pure mathematical consistency. And perhaps a comfortable death of carbon monoxide.

I have never seen anything really cool that is that vanilla. Try using opengl in some advanced case; use c++ for a serious task (and succeed), play with CUDA. Badass tech makes sacrifices and sometimes these sacrifices are exactly in some area that causes serious problems. It is like really good scotch; you have to come to it and just take it for what it is. Judge it because it fails to wipe your ass for you and you miss out on some really fucking awesome engineering.

I think that Rich should probably fix this. But if he doesn't it is a problem that I can trivially work around. I would *much* rather use clojure because I think it is a super fun system to play with and I know that I can work around anything it throws at me, no question. I did it with .NET, I have done it with c++, I can trivially do it with the JVM and clojure.

Abstractions have leaks. And the really good abstractions still have really painful leaks. Learn them, understand them, and move on. Save the judgement for someone who cares.

Chris

Why I love clojure

There is a fundamental fact about programming languages; or rather my interaction with them. I love the newb feeling.

This is a dangerous addiction because it doesn't take long before you never have the newb feeling; at most I have another couple years where I can find languages that fascinate me.

I just thought it was damn fun trying things out in the repl and just finding interesting things.

Somehow, c++ has lost that interest for me and I think I know why.

First off, it is just butt ugly looking. Templates look like shit; although I think they are cool as a compiler-extension mechanism.

Second off, you get problems like this. Today at work, working with a large piece of legacy code that I didn't write, a co-worker and I added something that made the program just crash. During a memory-reclaim operation (embedded game engine; thus it partially does its own memory management).

We went through the obvious possibilities; the most likely being if the object were deleted twice somehow. That wasn't the case, so we wandered around the code. Finally I decided to try changing the order of inheritance for an object with multiple inheritance. That fixed the problem.

Somewhere in the code, there are lines that reinterpret_cast something they should be using static_cast for. The project doesn't have dynamic_cast enabled so that is out of the question. That pisses me off but is only one of about 100 things about it that piss me off.

Anyway, the difference is that during multiple-inheritance, the actual pointer-value will change during an upcast or a downcast. This is because the way c++ objects are stored in memory and the the vagaries of v-table implementations.

This is, coincidentally, why the diamond of death is such a big deal in c++. You end up with two representations of the top of the diamond in memory. Thus:

     A
  B    C
     D

A would be in D's memory allocation twice. Thus if you did static_cast<A*>(static_cast<B*>(d_instance)) you would get a different answer than if you did static_cast<A*>(static_cast<C*>(d_instance)). Finally if you want to avoid all of this you can use virtual inheritance in c++. This looks like class B : virtual public A.

Then, however, access to a base-classes' data takes longer because there is an extra pointer in the middle.

All of these details distract you from getting your algorithm perfect or doing a very, very good design and thus I think that most c++ programs are fundamentally worse designed than a lot of programs in other languages.

You can really concentrate on only so much simultaneously. The more you are focusing on the details of an arcane language the less you are focusing on your algorithm and its fail-cases. Or the bigger picture in the sense of where this piece of code fits in the system; or how minimally you can accomplish the task.

The pain of refactoring c++ leads you to do an inordinate amount of up-front design which is always worse unless you are solving the same problem again which you never do.

Finally, TheEyeStrainCausedByLookingAtCamelCaseClassNames, CamelOrJavaCaseVariableNamesAlongWith half_of_the_language_using_underscores_which_no_one_uses_even_though_the_damn_language's_standard_library_is_written_with_it. Mix this with reading about 10 times the characters (literally). And what exactly does this mean?
(this->*item[index])(reinterpret_cast<SomeStupidClassName*>(*data)[4]);

And tell your coding conventions to fuck themselves. I can write better, more understandable code without conventions than you will ever touch with your 10000 lines of coding conventions; most of which have never been proven to improve the readability of sections of code. Every second you spend on coding conventions you would have spent on good or god-forbid great design and frankly, just pure bad design makes things hard to work with. Not code conventions. So unless you have spent the time going through thousands of programs all in different languages all with different coding conventions so you have the breadth and depth of knowledge to make an even semi-informed opinion on what makes code understandable to other people do not write a single line of a coding convention. Because frankly, you don't have any idea what you are talking about.

Man, now that is done with. I doubt I would hate code conventions as much as I do if I haven't worked in a language that requires so much goddamn typing because its base levels of abstraction are just too low to express the vast majority of concepts clearly or precisely.

It was brilliant for java and c# to take the route that they did and take the worst aspects of c++'s type system and continue them. If you want types, use good type inference. If you don't, use a dynamic language. Piss poor required static typing is just a waste of characters. Since every line you write is a liability, that leads code that is overly verbose, hard to refactor, and *requires* sophisticated editors to manipulate in some excuse for efficiency.

I have arrived at the point in my career where I would take slow, interesting, concise code over fast boring as hell and tedious code. I have never had an unsolvable optimization problem; and I have written mmx assembly, x86, used Intel's SSE and a bit of AMD's 3d now vector instruction toolkits, not to mention some insanely fast linear algebra for large datasets using CUDA.

Some things are just tedious and kind of suck. Most things can be done elegantly. Clojure makes doing the tedious no more tedious than it was going to be anyway and doing the elegant simple and interesting.

Chris

On the fact that camel case is stupid.

Capitalization is something I do as little of as possible. I write my name Chris, and I talk about NVIDIA. After that I hate it, its just that i makes you look like a little kid sort of like i like stickers and run on sentences. I like the capitalization of letters at the beginning of a sentence because my eye gets lost without it as I sometimes miss the period. I have tried to not capitalize things altogether and I had to stop.

CamelCaseNamesDriveMeCrazy. asDoJavaCaseNames,Really. dash-case-is-the-easiest-for-me, while_underscore_case_is_a_bit_worse_but_I_can_handle_it.

I-remember-my-grandfather-use-to-write-using-dash-case-when-leaving-notes.

"Anyone-caught-running-water-into-this-basin-will-be-in-trouble. Accidental-or-other-wise!" was something he wrote.

What about syllable-case?

ChrisToPherPeTerNuernBerGer

graphics file formats

It is an odd fact of life for 3d programmers that the file formats *really* suck.

The collada specification *just the spec* is a 3.7M pdf. It doesn't have a lot of pictures.

The collada xml schema is over 400K bytes and clocks in at 11,848 lines.

The FBX toolkit has a c++ library as a file specification. They can't be bothered to publish their file format, even thought it extremely regular and pretty damn well designed by my estimation. You can see it in ascii, but I wonder if the binary version is identical to the ascii or if there is something neat about it.

Then check out lib3ds, try checking out a maya file. Obj is ok if you need something fairly simple.

Supporting any one of these in an application takes a huge amount of work.

Out of all of these I only really know collada. I did a 3ds importer a long time ago. One time I reverse engineered the excel .xls format going from partial documentation and the source code to OpenOffice's source code. Even with all of that, it took a *lot* of effort. Lots of hex dumps. Anyway...

Collada is the best out of all of them in my opinion. I bet I am one of a very very few number of knowledgeable graphics developers who believe that, but it is. They all suck, but at least on has an open development process along with a few large companies pushing it. It isn't a game engine data format; xml is big, verbose, and extremely slow to parse compared to fairly trivial binary file formats. It isn't the best designed piece of software I have ever seen, either. It isn't modular nor entirely consistent (like something of that size could be consistent, but anyway); in some places it is way overly generic and arbitrary (animations. There are at least 5 different ways to store bezier and linear key data. 5. FX composer, 3dmax, maya, and another 3d editor who's name escapes me now because they aren't sold in the US all do things them differently. It doesn't have nearly enough examples of what could be considered the right way to do things; thus everyone has just rolled their own.

But, it is something people can rally behind and we can all use it. I think the way collada does transforms is really really good. They are the source of a lot of problems, but you can do anything with them. Really goddamn good. I spent a huge amount of time getting them to work, but I think they are cool. I also think the way they store geometry data is pretty cool and lends itself perfectly to using opengl efficiently (vbos). I am not a huge fan of its shader sections; the interaction between shaders and scenes (using evaluate_scene tags) kind of annoys me. I just don't think of shaders, be they a special material on an object or a multi-pass effect in the same model and it irks me.

I like collada's extension mechanism but it is clunky. Certain really odd situations happen with different interpretations of the specification.

I wish they had used certain elements of xml schema and don't things a little differently in others. For instance, they don't break up logically different spaces using sub-schemas and xls-include. I don't know if this is for technical reasons, but it makes interpretating the entire specification tough because you *have* to look at a lot of stuff.

Why there isn't collada-geometry that includes collada core, why isn't there collada-animation in a completely different schema that includes common bits? Why doesn't the file start with their extension mechanism and use that to make pieces more modular? Perhaps you can't really have a unified ID-space if you are using different schemas? It may break schema verifiers; which for something like collada are particularly useless as it does lots of outside-schema stuff anyway, and certain large vendors produced files that didn't pass verification as it was.

I believe the FBX file format has superior architecture but it isn't open and it has weird characteristics that really irk me. I wish it had collada's geometry section, and for that record the way collada at least partially allows a good design for shaders and effects blows FBX out of the water. It starts with very generic design (essentially objects with arbitrary properties) and uses convention to and their API to ensure consistency. The details of the data in the format aren't as smart as Collada, however, and that kills it for me.

They both are light years better than 3ds files. I bet maya is pretty good design as that app is really in a league of its own with regard to really amazingly smart design. Also, for the record, Adobe can do some unbelievably cool things with applications. After Effects, even after all of these years is an amazing application. Their plugin API, while tedious and sometimes very poorly thought out, is documented with humor and looking at some of the things they allow is quite enlightening as to the application's internal architecture.

This is perhaps the weirdest thing I have admitted for a long time but file formats are really fascinating to me. API's almost always bore me or piss me off. But for some reason file formats are interesting; especially old ones. The excel file format is really damn cool, and some of the things that the microsoft applications could do was legitimately goddamn tough to do in binary. Things like in-place editing, where the app could run directly from the file without ever loading the entire file into memory at once and could write to the file in the same locations without growing its size (or growing it in a very generic way). Meaning the application could memory-map the file and then instantly use structures with no further loading, theoretically. It could also perform edits to this file while simultaneously growing its memory. This isn't shit tricks done with malloc or new; this is really good, consistent and darn hard engineering challenge that would affect the design of your entire application. On the other hand, the applications take forever to load files now and you have to wonder. Meanwhile KOffice tends to insta-load apps so fast it can be somewhat shocking.

I hate microsoft office; but I believe that in odd corners of it you find some of the most amazing engineering I have ever seen. Excel is an application that has yet to be equalled (or even close) by an opensource alternative. The best open source office suites just don't even compare to Microsoft's tools in terms of quality, features, or unified, consistent design. I guess I just hate office software in general as just the look of it forces me to puke instantly while seeing images of baby Jesus crying.

/ramble

Chris

Project update

Finished the first few sections of the refactoring of the project. I believe I will eliminate about 1/4 the code through eliminating dead code and careful refactoring as well as structure the code in a way that doesn't require another dev to know everything all at the same time.

I am giving a demo of its functionality to other NVIDIA people Friday so I can show off some of the really cool features. I noticed that an artist decided to attend the demo, which means I need to think a lot harder about the presentation and the audience. I intend to give another demo to the functional programming languages group later.

I want to present a couple of things. Thinking about it now, I would like to be able to present each feature in a way that everyone at the demonstration, regardless of technical ability, can really understand why it is important.

First off, I want a few of the more impressive features of the app clearly demonstrated. The live shader editing is going to be a hit, as well as perhaps some of the user interface details. This presentation is certainly not about the graphics, however. When this super clear, thorough, and much better than I could have ever written it in my entire life tutorial is working then I will have some eye candy.

I want to talk about the architecture behind the system, especially about the really heavy usage of threads. I guess I need to talk about software transactional memory and exactly why writing really safe multithreading code is much easier with this paradigm. Which means I need to explain functional datastructures, structural sharing, and software transactional memory.

I want to show off the repl a little bit. I have emacs very lightly pimped out and I know how to use the repl, so that should be fun. Having a repl may beat having an entire IDE. This is assuming you design your systems to be run from the repl. Not sure exactly how to prove this, but I just want to make the differences in the workflows very apparent.

So why is the repl so much better?

Having the repl is like having your debugger and your source code editor always running and always checking things out. You can weave in and out of editing and checking results/debugging very fluidly; this makes the distinction between writing and testing or debugging code disappear. Furthermore, being able to write small pieces of things and test them out immediately quickens your adoption of more advanced language features. Finally, you tend to test functions quite thoroughly from the repl. It is much quicker to test them and just look at their output from the repl than it is to write a unit test, run through the failures and debug the unit test.

This had the unfortunate side effect of eliminating the testing codebase I usually write; this I am not quite as pumped about as it means another dev will need to be *much* more careful about what they are doing. Well, win some and lose some.

I wonder how hard it would be to have emacs connect up to another running process? Emacs always starts the process; it can't be all that difficult to have emacs look for a connection at a given port. I wouldn't think it would be too much code to write in your app to have it open up a connection and start whatever clojure-swank starts...

Man, clojure is cool.

Chris

First Clojure Project

Finished my first clojure project yesterday, at least finished functionally.

I wrote a small program where you have some 3d graphics running and you can change the anti-aliasing settings, the arguments to the glsl shaders, and a few other things.

It main uses clojure, JOGL and swing. It is running doing 4x antialiasing of the screen on my system and using about 35% of one of my two cores. Most of that time is in opengl or swing, I am not sure which. I have figured out that for this program, java 1.6 significantly outperforms java 1.5. This may be due to a much faster swing rendering implementation that is itself rooted in opengl, or the fact the the 64 bit java implementation is faster, or a number of things.

When I have thought of a better name for it I will move it to a new github repository and people can try it out. You would need to install java, JOGL, clojure, and clojure-contrib.

So, here are some of the harder issues encountered doing this:

1. Figuring out which windowing library to use was horrid because I tried QT first. It hung or crashed for days; I tried installing different jvms and all manner of things. Mixing QT with JOGL and clojure really was a waste of time; QT-java, for the mac, just isn't ready for primetime. This was the only real issue that really wasn't fun to solve.

In implementation and debugging (these were fun to figure out).

2. Laziness confused me several times. One time it was classes, I called map and didn't use the return value. Thus nothing happened; I wanted the side effect of writing out to a file. Another time I use a lazy list while doing opengl stuff. I tried to look at this list in the REPL. This caused the entire jvm to crash with a bus error. This is because the repl and opengl run in different threads and calculating the result on demand was causing the list to be evaluated in the repl thread. This was the type of bug that, when you figure it out, you know just marked a significant step in your path to functional zen.

3. Swing layout systems. Every goddamn time I use these I spend forever on at least one stupid problem. GridBagLayout-4-eva. At least with a repl I could literally change the code, hit return and see the results. It really was kind of interesting.

4. REPL madness. I managed to get the program into all sorts of weird states by reloading files in the repl. I produced another bus error, I got things to hang. Lots of stuff. I would sit there an change opengl commands telling all sorts of stuff, hit return, and laugh as it did something weird. I would exit, build the project jar, restart and the program would behave differently because I hadn't loaded a file that I had edited or something along those lines.

5. java.nio.Buffer. There was a bug that took me a long time to solve when I switched to vertex buffer objects. The way you pass these to JOGL is using the newer java.nio.FloatBuffer (or CharBuffer or ShortBuffer...). Anyway, I would hit render and nothing would happen. At first I thought it was perhaps my vertex shader. Then I was thinking it was in the binding the vertex buffer object to this particular vertex shader property. Still nothing. Finally *finally* after working on this for like 2 hours, I looked at the API for java.nio.Buffer. It has a flip function. I deduced out that when you fill a buffer from an array, its position member variable gets set to the end of the buffer. The flip function sets this variable back to 0. This set it up for a read operation somewhere on the JNI side of things.

Super, super brutal mistake caused by a few things. First off, when you call the JOGL call you pass in an explicit byte-count argument. I figured if JOGL had this information, it should be able to take the buffer and just make it work. Second off, I didn't even realize that a java.nio object *had* flip function. Third, I have never used anything with a flip function so the initial read over the API didn't help anything out.

6. glVertexAttribPointer takes *byte* arguments. I was giving it an array of floats, I figured its arguments would be float sized since it knew the buffer that was bound at the time of its invocation and thus knew the datatype in the buffer. After getting extremely bizarre results for another couple hours I figured this out. This was unfortunately when I was hooking up the antialiasing code which also renders to a multi-sample fbo, then downsamples this multi-sample fbo to a single (non antialiased) sample fbo and then you finally render this to the screen. Get it? So there were a lot of links in the chain that could have failed. Oh yeah, and I had a to render to a fullscreen quad for the final step to the screen so I had this other piece of the chain that was failing for a while.

7. The general problems related to running a realtime rendering system. These include: Swing completely destroys the opengl render context every once in a while on resize. I wasn't initially planning to handle this condition because it would traditionally only happen if you ran a screensaver or something like that. This isn't a rip on swing; I understand why they render to pbuffers and thus why they have to reallocate them sometimes. The java debug opengl interface, instead of only throwing an exception once upon error and resetting, throws an error for every single opengl call after the error is made. Thus I would get exception stack frames printed to the repl at 60 frames/second.

The upside of most of the problems of 7 is that I built a much more robust rendering platform much earlier than I was planning. So it automatically reloads files that have been kicked out of the system, rebuilds vertex buffer objects, is generally pretty tough.


OK, so enough about the problems. What did I build? What did I get out of the experience?


I proved clojure's viability, at least at some level, for doing 3d graphics. They are certainly possible. As swing gets faster, they will get more possible. Plus, not all applications need to update at 60 frames a second all the time. Additionally, my usage of clojure is still quite amateur. As I learn more about the language and swing, I may have significant opportunities to make things cooler.

The app is really cool, tight, and small. It has an application log, a 3d view, and an inspector palette where you can check out the properties of things you are looking at. You can click on a shader file and (at least on my mac) it will open the file in an external editor and start listening for changes. Any time the file changes, it attempts to reload the file and shove it back into the program. If it works then you get a new program. If it doesn't it saves the result, deletes intermediates and prints the error log to the application log. Thus you can sit there and tweak shaders till you are dizzy and the app will continue to display reasonable stuff and tell you exactly why the shader didn't load.

You can switch the antialising on or off, which actually changes a bit of the rendering pipeline, and you can choose the amount of aa you like. My laptop, for example, only supports 4x antialiasing. Thus if you select 16x it will try that. Failing 16x oversampling, fall back to 8x and try that. It will continue doing this until it finds antialiasing that is supported. This is supported in the graphics library; you can pass it a sequence of surface specifications and it will try each one until it finds one that is valid.

The docking panel framework I found on the web that is OK. Not great, but not horrible. It wouldn't work for a commercial application but it is like Christmas for a opensource of shareware application. Plus it is LGPL'd. Anyway, you can dock, undock, and do all manner of stuff with the windows.

The application is built on a heavily threaded architecture. OpenGL can only run in one thread, so that is capped. There is a large thread pool for blocking IO and a smaller thread pool (the number of processors) for CPU-bound processing. This is taken care of using clojure's agents which I bastardize to do what I want. I don't use them the way they were intended; I just use their send and send-off commands.

So, for instance, lets say you want to load a file. All files are md5'd upon load so that I can avoid doing anything redundant with large pieces of data. Lets say this file is a vertex shader that is used in one or more programs.

An IO thread loads the file into a byte buffer. It then hands this byte buffer to a CPU thread to do the md5. Finally, the rendering system picks up this new buffer (during the render loop) and tries to make a shader with it. Should it succeed, it finds all programs that included the old shader and attempts to re-link the programs with the new shader and its old counterpart (the shader could have been either vertex or fragment). If it succeeds it replaces the old program with the new one. At each point it logs something.

The log data gets shoved onto the a CPU thread where it is split into lines, has module and message type appended to the front. The application log messages list gets appended to with the new log data.

There is another thread that every so often, perhaps 3 times a second, checks to see if the written log messages are different than the application log messages. If they are it copies the app log messages to the written messages list and fires off a thread that takes the written messages list and builds a large string of them all concatenated together. It then fires off yet another thread using SwingUtilities/invokeLater that does the actual text setting on the log window object.

I didn't want logging to block whatever thread is doing the logging. I didn't want the log window to be appended to very often because this is an extremely expensive operation; especially when the log is long (it is capped at around 1000 lines right now in a clever, lazy fasion using 'take'). I wanted any real work to be done in a CPU thread and not in the logging thread or the UI event thread. Finally you shouldn't access swing components outside of the swing thread if you can help it.

So, the point is that I am using threads like they are growing on trees. Because I have two cores and I know for a fact the second core is usually doing absolutely nothing. This is nothing compared to what will happen over time as i get more and more functionality; I love threads and continuation style threading.

Now, I have all the functionality I want and the application is damn stable and responsive. If it runs, I bet you can't crash it. I can say this because it has very fixed inputs and I have tested every one of them exhaustively, which is feasible in this case.

Lets talk about repl-based development. Lets take my FBO (frame-buffer-object) implementation. Starting from zero, I first write a function that pushes a closure onto a list that gets run on the render thread. It waits for this function to finish (using a CountDownLatch), and then returns the result to the repl *just as if the function was synchronous*. Even though it happened on the render thread.

Next I write a function that creates an FBO. I test this function extensively with the repl right then and there. I pass in nonsense, I passing in large numbers, I find out all about the failure modes of my FBO allocation implementation (of which there are a few). Now I do the same for a simple FBO delete function, chained after an allocate function. Next I test managing maps of named FBO objects so you can create an FBO and refer to it by name to get it later. For the most part, all of this is done without shutting the program down.

In 3d-graphics, this is pretty hard to get right. But designing your code to be run from the repl has the same effect as designing it to be unit tested; it is just a lot tighter and easier to mess with.

Anyway, I used this technique for designing the user interface of my program. Swing layouts really suck to get right. They are time sinks; but a repl and a good testing strategy *really* help things out to speed things up.

OK, so you know what hurt and you know what I thought was cool. Now the next steps...

Here are the famous (and correct) steps to creating good systems:
1. Make it work. Be goddamn sure that you understand the problem.
2. Make it elegant. <== we are here
3. Make it fast. (probably doesn't apply yet).

Now, how do we make it elegant?

1. Remove as much as possible. I call this the code compression stage, although a better way to state it would be the code evaporation stage. This is where you study the language, the code, and really think about your algorithms and the way they are implemented to see if you can think of more concise (but not overly clever) ways of doing things. This is also there I will try to significantly improve the modular decomposition of my code; write better utilities and break code up into generic pieces. This is the best part; where you take a rough sculpture and make it truly a piece of art.

2. Ensure API exposure is consistent. This means that there are consistent names and overall structure to the system. I am doing very poorly on this point right now because I was learning the language while I was writing code. This is also where you separate public API's from internal APIs and document public api thoroughly.

3. Attempt to match the idioms of the language. I used underscores throughout my program because I like them and I didn't realize I wasn't supposed to; I will replace these with dashes because clojure uses dashes. You should also look at your usage of standard library functions and try to make sure you understand how they are supposed to be used. Remove any functions you wrote that are replacements for standard library functions; etc. Go through the clojure libraries and try to match the naming and design conventions of the major packages.

When I code, I do whatever is expedient because I want to see something work. But after I see something work, I want to make it really nice. The above steps are my standard steps and really they just facilitate me thinking about what I did in a very thorough but still abstract manner.

Chris

Goddamn ant macosx xml problem

[ chrisn chris-nuernbergers-macbook-pro ~/dev/editor/lambinator ] ./buildme.sh
Buildfile: build.xml

init:

compile_lambinator:
     [java] Compiling lambinator.experiment to /Users/chrisn/dev/editor/lambinator/classes
     [java] Compiling lambinator.ui to /Users/chrisn/dev/editor/lambinator/classes
     [java] java.lang.ExceptionInInitializerError (ui.clj:1)
     [java]  at org.apache.tools.ant.taskdefs.ExecuteJava.execute(ExecuteJava.java:194)
     [java]  at org.apache.tools.ant.taskdefs.Java.run(Java.java:747)
     [java]  at org.apache.tools.ant.taskdefs.Java.executeJava(Java.java:201)
     [java]  at org.apache.tools.ant.taskdefs.Java.execute(Java.java:104)
     [java]  at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
     [java]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     [java]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     [java]  at java.lang.reflect.Method.invoke(Method.java:585)
     [java]  at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:105)
     [java]  at org.apache.tools.ant.Task.perform(Task.java:348)
     [java]  at org.apache.tools.ant.Target.execute(Target.java:357)
     [java]  at org.apache.tools.ant.Target.performTasks(Target.java:385)
     [java]  at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1329)
     [java]  at org.apache.tools.ant.Project.executeTarget(Project.java:1298)
     [java]  at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
     [java]  at org.apache.tools.ant.Project.executeTargets(Project.java:1181)
     [java]  at org.apache.tools.ant.Main.runBuild(Main.java:698)
     [java]  at org.apache.tools.ant.Main.startAnt(Main.java:199)
     [java]  at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
     [java]  at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
     [java] Caused by: java.lang.ExceptionInInitializerError (ui.clj:1)
     [java]  at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:2684)
     [java]  at clojure.lang.Compiler$BodyExpr.eval(Compiler.java:3631)
     [java]  at clojure.lang.Compiler.compile(Compiler.java:4564)
     [java]  at clojure.lang.RT.compile(RT.java:362)
     [java]  at clojure.lang.RT.load(RT.java:404)
     [java]  at clojure.lang.RT.load(RT.java:376)
     [java]  at clojure.core$load__4557$fn__4559.invoke(core.clj:3427)
     [java]  at clojure.core$load__4557.doInvoke(core.clj:3426)
     [java]  at clojure.lang.RestFn.invoke(RestFn.java:413)
     [java]  at clojure.core$load_one__4520.invoke(core.clj:3271)
     [java]  at clojure.core$compile__4563$fn__4565.invoke(core.clj:3437)
     [java]  at clojure.core$compile__4563.invoke(core.clj:3436)
     [java]  at clojure.lang.Var.invoke(Var.java:327)
     [java]  at clojure.lang.Compile.main(Compile.java:52)
     [java]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     [java]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     [java]  at java.lang.reflect.Method.invoke(Method.java:585)
     [java]  at org.apache.tools.ant.taskdefs.ExecuteJava.run(ExecuteJava.java:217)
     [java]  at org.apache.tools.ant.taskdefs.ExecuteJava.execute(ExecuteJava.java:152)
     [java]  ... 20 more
     [java] Caused by: java.lang.ExceptionInInitializerError
     [java]  at com.trolltech.qt.QtJambiObject.(QtJambiObject.java:57)
     [java]  at java.lang.Class.forName0(Native Method)
     [java]  at java.lang.Class.forName(Class.java:164)
     [java]  at clojure.core$import__3583.doInvoke(core.clj:1600)
     [java]  at clojure.lang.RestFn.applyTo(RestFn.java:142)
     [java]  at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:2679)
     [java]  ... 39 more
     [java] Caused by: java.lang.RuntimeException: Loading library failed, progress so far:
     [java] Unpacking .jar file: 'qtjambi-macosx-gcc-4.4.3_01.jar'
     [java] Checking Archive 'qtjambi-macosx-gcc-4.4.3_01.jar'
     [java] 
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.loadNativeLibrary(NativeLibraryManager.java:428)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.loadQtLibrary(NativeLibraryManager.java:352)
     [java]  at com.trolltech.qt.Utilities.loadQtLibrary(Utilities.java:137)
     [java]  at com.trolltech.qt.Utilities.loadQtLibrary(Utilities.java:133)
     [java]  at com.trolltech.qt.QtJambi_LibraryInitializer.(QtJambi_LibraryInitializer.java:53)
     [java]  ... 45 more
     [java] Caused by: java.lang.RuntimeException: Failed to unpack native libraries, progress so far:
     [java] Unpacking .jar file: 'qtjambi-macosx-gcc-4.4.3_01.jar'
     [java] Checking Archive 'qtjambi-macosx-gcc-4.4.3_01.jar'
     [java] 
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.unpack(NativeLibraryManager.java:365)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.loadLibrary_helper(NativeLibraryManager.java:434)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.loadNativeLibrary(NativeLibraryManager.java:423)
     [java]  ... 49 more
     [java] Caused by: javax.xml.parsers.FactoryConfigurationError: Provider org.apache.xerces.jaxp.SAXParserFactoryImpl not found
     [java]  at javax.xml.parsers.SAXParserFactory.newInstance(SAXParserFactory.java:113)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.readDeploySpec(NativeLibraryManager.java:496)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.unpackDeploymentSpec(NativeLibraryManager.java:521)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.unpack_helper(NativeLibraryManager.java:389)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.unpack(NativeLibraryManager.java:360)
     [java]  ... 51 more
     [java] --- Nested Exception ---
     [java] java.lang.ExceptionInInitializerError (ui.clj:1)
     [java]  at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:2684)
     [java]  at clojure.lang.Compiler$BodyExpr.eval(Compiler.java:3631)
     [java]  at clojure.lang.Compiler.compile(Compiler.java:4564)
     [java]  at clojure.lang.RT.compile(RT.java:362)
     [java]  at clojure.lang.RT.load(RT.java:404)
     [java]  at clojure.lang.RT.load(RT.java:376)
     [java]  at clojure.core$load__4557$fn__4559.invoke(core.clj:3427)
     [java]  at clojure.core$load__4557.doInvoke(core.clj:3426)
     [java]  at clojure.lang.RestFn.invoke(RestFn.java:413)
     [java]  at clojure.core$load_one__4520.invoke(core.clj:3271)
     [java]  at clojure.core$compile__4563$fn__4565.invoke(core.clj:3437)
     [java]  at clojure.core$compile__4563.invoke(core.clj:3436)
     [java]  at clojure.lang.Var.invoke(Var.java:327)
     [java]  at clojure.lang.Compile.main(Compile.java:52)
     [java]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     [java]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     [java]  at java.lang.reflect.Method.invoke(Method.java:585)
     [java]  at org.apache.tools.ant.taskdefs.ExecuteJava.run(ExecuteJava.java:217)
     [java]  at org.apache.tools.ant.taskdefs.ExecuteJava.execute(ExecuteJava.java:152)
     [java]  at org.apache.tools.ant.taskdefs.Java.run(Java.java:747)
     [java]  at org.apache.tools.ant.taskdefs.Java.executeJava(Java.java:201)
     [java]  at org.apache.tools.ant.taskdefs.Java.execute(Java.java:104)
     [java]  at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
     [java]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     [java]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     [java]  at java.lang.reflect.Method.invoke(Method.java:585)
     [java]  at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:105)
     [java]  at org.apache.tools.ant.Task.perform(Task.java:348)
     [java]  at org.apache.tools.ant.Target.execute(Target.java:357)
     [java]  at org.apache.tools.ant.Target.performTasks(Target.java:385)
     [java]  at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1329)
     [java]  at org.apache.tools.ant.Project.executeTarget(Project.java:1298)
     [java]  at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
     [java]  at org.apache.tools.ant.Project.executeTargets(Project.java:1181)
     [java]  at org.apache.tools.ant.Main.runBuild(Main.java:698)
     [java]  at org.apache.tools.ant.Main.startAnt(Main.java:199)
     [java]  at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
     [java]  at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
     [java] Caused by: java.lang.ExceptionInInitializerError
     [java]  at com.trolltech.qt.QtJambiObject.(QtJambiObject.java:57)
     [java]  at java.lang.Class.forName0(Native Method)
     [java]  at java.lang.Class.forName(Class.java:164)
     [java]  at clojure.core$import__3583.doInvoke(core.clj:1600)
     [java]  at clojure.lang.RestFn.applyTo(RestFn.java:142)
     [java]  at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:2679)
     [java]  ... 39 more
     [java] Caused by: java.lang.RuntimeException: Loading library failed, progress so far:
     [java] Unpacking .jar file: 'qtjambi-macosx-gcc-4.4.3_01.jar'
     [java] Checking Archive 'qtjambi-macosx-gcc-4.4.3_01.jar'
     [java] 
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.loadNativeLibrary(NativeLibraryManager.java:428)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.loadQtLibrary(NativeLibraryManager.java:352)
     [java]  at com.trolltech.qt.Utilities.loadQtLibrary(Utilities.java:137)
     [java]  at com.trolltech.qt.Utilities.loadQtLibrary(Utilities.java:133)
     [java]  at com.trolltech.qt.QtJambi_LibraryInitializer.(QtJambi_LibraryInitializer.java:53)
     [java]  ... 45 more
     [java] Caused by: java.lang.RuntimeException: Failed to unpack native libraries, progress so far:
     [java] Unpacking .jar file: 'qtjambi-macosx-gcc-4.4.3_01.jar'
     [java] Checking Archive 'qtjambi-macosx-gcc-4.4.3_01.jar'
     [java] 
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.unpack(NativeLibraryManager.java:365)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.loadLibrary_helper(NativeLibraryManager.java:434)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.loadNativeLibrary(NativeLibraryManager.java:423)
     [java]  ... 49 more
     [java] Caused by: javax.xml.parsers.FactoryConfigurationError: Provider org.apache.xerces.jaxp.SAXParserFactoryImpl not found
     [java]  at javax.xml.parsers.SAXParserFactory.newInstance(SAXParserFactory.java:113)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.readDeploySpec(NativeLibraryManager.java:496)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.unpackDeploymentSpec(NativeLibraryManager.java:521)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.unpack_helper(NativeLibraryManager.java:389)
     [java]  at com.trolltech.qt.internal.NativeLibraryManager.unpack(NativeLibraryManager.java:360)
     [java]  ... 51 more

Solution:
sudo mv /usr/share/ant/lib/xercesImpl.jar /usr/share/ant/lib/xercesImpl.jar.back

Don't ask me how I figured that one out.

Clojure Projects

I don't really know how to start this other than to look at various ways you can create functionality in a clojure project and attempt to categorize them in some productive way.

The first I want to talk about is the repl. Hopefully you have had the pleasure of working with a good repl setup (SLIME's is the best I have ever used, have you seen better?). REPL stands for read-eval-print-loop. It looks like a command prompt and you type in some stuff and immediately see what the result is. This is similar to when you break in a debugger and you can both analyze values *and* do edit and continue.

I am going to out on a limb here and state that the REPL is the fastest way to add new functionality. You type in a couple statements and you see, right after you type them if it worked or not. This is like Christmas. It is a fucking blast if you have never used it; trust me. It feels weird at first but you just have to use it for a couple days and you will be amazed at how much stuff you get to work. The rate you can add functionality to a new system is directly related to the rate that you can get feedback as to the correctness of this functionality. Typed languages use the static typing system to give you a little faster feedback but nothing means anything until the bits are actually moving; thus the repl is the best. It is more fun than diagnosing hindley-milner errors anyway.

OK. The problem with the repl is that you would have to type in your program all the time. So you have files where you can save the information that you can dynamically load into the repl. You can use either (load-file "fname") or you can use slime-compaile-and-load-file from emacs. This is almost as much fun as the repl and a lot more repeatable. The file is still dynamically loaded but at least you can replace sets of functionality at one, easily.

Next we can compile a file and put it into a jar file; or more generally some compilation unit. These are *not* dynamically loadable to my knowledge; thus if you want a program to run for a long time you will load the jar files once and then you are off. Hopefully the program is completely correct and never needs runtime updates.

Jar files are more easily reusable and distributable than stand alone text files. They also protect your IP to some extent although I really don't give a shit about that.

So we have three levels of adding code to the system, in order of ease of mutability. Repl, repl-loaded, and jar'd. So how do we set up a system that makes all this cool?

Well, lets say we have a set of .clj files. We want an ant project that will compile and jar up all of them. I will set this up in a second.

Now, we have to have this jar file path in our classpath somehow. You can either add it via (set-classpath) or you can add it to the java "cp" startup argument.

Now we find some function is wrong in this jar file. Can we dynamically replace this function in a running system? Well, assuming it is reference via 1 level on indirection then yes, we can. This is what the whole environment and "defn" system gets you; I am assuming. But this requires a test.

Basically, if I can create a jar file and then replace some of its functionality in the running system then I am golden; I can have fast, precompiled code *and* I can update the system and try out fixes and new ideas dynamically. I believe, due to the way that clojure's Vars work that I can get this stuff running. But, hope is not a strategy so lets try some of this out.

The first step is to make a jar file out of some clojure file. In the simplest case, I have a clojure file and has a single function in some namespace, and then I jar it up using ant. Next I add it to my classpath and load it into the repl, checking that it works. Finally I replace said function somehow using the repl.

There is one guaranteed caveat that was there in common lisp and it is here too. If you create a closure dynamically this won't be replaced if you load a new file. Thus you need to figure out what level of dynamic functionality you want and avoid closures where necessary, or create closures that immediately call into the namespace vars.

So here we go. Into the lambinator directory and create a subdirectory called src/experiment

OK, copied clojure-contrib's jar file (after actually getting it to produce useful output, you have to tell ant where clojure.jar is). Got the dynjar.clj file compiling into a class directory and having classes shoved into a jar file. At this point I am now as good at clojure as I ever was at java.

I have to update my ~/bin/clojure script to add this jar file to my classpath. I know this will change per project but emacs needs this command; I could perhaps have project-specific .emacs files but I am really not that ambitious.

In my test file, I have one function that returns an integer, and another function that returns the result of the first. Now the test is that if I load the jar file and mess around with the repl I should be able to get the second function to return a new value by replacing the first.

So:
a -> 5
b -> a -> 5

In repl, I will change a to return anything but 5. If this works, then I can dynamically update my running program with new stuff!

Lets see what happens:

Hell happened. Jar hell.

load-file does the equivalent of import and require on all the objects in the file. This was not quite what I expected.

I will update this post later when I have much more information.

Getting emacs working

Looked through the clojure-mode source for some good stuff:

unfortunately, clojure-load-file doesn't seem to work due to:
comint-get-source, and then after I start slime *inferior-lisp-proc*.

slime-load-file worked just fine. I wonder if there is a slime-load-all-out-of-date-files. That would be damn useful.

Some shizzle you should know

OK, so something I use a lot in other editors is bookmarking facilities.

So, in emacs we have two functions that I really care about: bookmark-set and bookmark-jump.

So, the idea is to set a point in a given file you know you will need later. You name these so they are unique.

A bookmark is a fully named entity, so these commands ask you for a name.

set: C-x r m

jump: C-x r b

The next thing I use all the time are registers if I have access to them. Basically, I want to select a bit of text and shove it into some named variable I will reuse in different contexts. Registers in emacs are single-character named entities you can put text (and other things) into and get it out of.

These commands will ask for a single character register name to place information into or get it from.

(select region then..)
copy: C-x r s
paste: C-x r i

In general, try to get by without using copy/paste. It *really* is the work of the devil. In any form.

OK, down to the basics. Remember this is aquamacs...
aquamacs-isearch-forward A-f
aquamacs-repeat-isearch , A-g
aquamacs-repeat-isearch-backward , A-G
isearch-backward C-r
isearch-forward , C-s


Don't forget basic navigation, either:
C-p
C-n
C-b
C-f

Finally, I put these two lines in my .emacs file.

(global-set-key [f4] 'slime-compile-and-load-file)
(global-set-key [f3] 'find-name-dired) //This one is *super* useful in larger projects

Trying out the various slime functions, they don't appear to work very well. The awesome ones are 'slime-who-calls and stuff like that; but anyway...

Well, we have hit the point of somewhat diminishing returns. Thus in the next post we will look at the next steps of a non-trivial clojure project.

Clojure App dev, step 1 -> repl-QT-repl-QT

The goal of this step is to ensure you can start a qt frame from the repl, switch to qt's event handling (thus leaving the repl) but get back to the repl using a button.

This is critically important; I need to be able to run the application and then update it dynamically *without* restarting anything. I can't tell you how many times, while debugging something like mouse picking or moving objects around in 3d space I had to break into the debugger, then check out what is going wrong. Next I would make an attempt to fix, recompile, reload the presentation and then repeat. Over and over and over and fucking over again. And again. What I would like is that I can see a given function isn't working correctly; I just update the function definition *while I am running the editor*. This is goddamn important. With this working complex, really useable features are much easier to get working correctly.

OK, so here we go!

It should look like:

(start from repl)
open qt frame with one button, return to repl.
switch to qt's event handling thus effectively leaving the repl
push a button and return to repl *without* closing frame
add a menu item to the existing frame, thus ensuring we can mutate the datastructure and get a good update
exit main frame; thus going back to repl.
see what the state of the main frame is; we may be able to open it

This relies on one key assumption: that I can exit qt's event handing without closing the window. I believe I can but I don't know.

If this assumption doesn't hold then I can always store, latently, the commands for creating the UI. Then when I want to repl around with shit I can destroy the entire UI, update the commands to add new elements, and re-create. This is heavy-handed and rude but it might work. It will remove perhaps 30-40% of the functionality that want, however so I will really try to avoid this.

Reading the QT documentation, it looks like you call QApplicationCore.exec() to get things going and then you can call QApplicationCore.exit() in order to return from the event loop. So your return-to-repl button should just call exit, perhaps. There is also a processEvents call that you could call. Essentially, you could loop over calling process events and have your button set a global that tells you to stop. But if exit works then that is the sweetest.

Simple clojure QT example

This should get me started. There is something really weird when you are learning a new language. You don't know how to create a hash, or vector, and it seems every character you type is wrong. It takes a couple days before you (or perhaps just I) get things going. On the other hand, this level of comfortability with the language has benefits. You focus far more on algorithm than details; this leads to better code.

For example, if you lived in C for all your life and was asked to do something one of the first things you would visualize or plan out would be the memory access system because the most complex part of most C programs is based around memory handling; at least as much as what the program is meant to do. It takes a few other languages with garbage collection before you begin to think at a higher level and then work down to the memory level if you have to.

Side tracked:

BillC figured this all out first
This too. It hasn't worked for me yet, though

Well, this blows. I have been trying to diagnose an error for quite some time now, it looks like:

[Thrown class java.lang.ExceptionInInitializerError]

0: com.trolltech.qt.QtJambiObject.(QtJambiObject.java:57)
1: java.lang.Class.forName0(Native Method)
2: java.lang.Class.forName(Class.java:169)

Checking shit out, it isn't immediately clear what is going on. Time to research how to debug this error....

The demos run just fine, so I know it is possible to run applications. Trolltech -helpfully- included a binary starter to the demos so I can't easily see what is going on.

Holy shit that hurt!

Why God Why Doesn't It Work?!?1!1?

**Don't use JDK 6**

Nice, 2 hours of fucking around with java madness. I finally figured out the magic google code that would find the problem exactly:

java.lang.UnsatisfiedLinkError libQtCore.4.dylib

So I need to amend my other post.

Some more things are working. It appears that QApplication.exit actually closes all open widgets. This isn't exactly what I want. So I will just call process events from a loop and have a button set a variable to break out of the loop.

Thus I would like to be able to define a variable in the namespace that is something like "process_events_running" and have the single button in my application set it to false. Then I will provide a custom exec function that calls process in a loop checking that variable.

The simples way to do this would be a closure. I don't immediately see how to do this, but I remember rich hickey stating that lambdas are "ICallable" or something like that. I know that the signal system in QTJambi uses reflection, so if I just pass in a closure as the "this" argument to connect, and pass in the function named call on the icallable I may be able to get somewhere. The interface is Callable:

user> (lambda `(println "hello"))
; Evaluation aborted.
user> (fn [] (println "hello"))
#
user> (set x (fn [] (println "hello")))
; Evaluation aborted.
user> (def x (fn [] (println "hello")))
#'user/x
user> (. x call)
hello
nil
user> x
#
user> (x)
hello
nil
user>

Still not finished, but I need some food.

OK, nice microwave pizza and I am back.

So, passing in a closure to connect works fine. I do it like this:

(defn exec []
(def exec_var 1)
  (while
(== exec_var 1)
    (QApplication/processEvents)))

(defn create_app_frame []
(ensure_app_init)
 (let [app (QApplication/instance)
       button (new QPushButton "Go Clojure Go")]
   (.. button clicked (connect (fn [] (def exec_var 0)) "call()"))
   (doto button
     (.resize 250 100)
     (.setFont (new QFont "Deja Vu Sans" 18 (.. QFont$Weight Bold value)))
     (.setWindowTitle "Go Clojure Go")
     (.show))
   button)) ;return the button for further reference

Now I need things to sleep because I am chewing up CPU by calling process events over and over again. There is a hasPendingEvents call; so now all I need is how to make the system sleep in a platform independent way. Java has Thread.sleep in the language and that is that.

I didn't mess with the datastructures of the frame yet (mainly because I just have a button). But I have a QT app, running from the repl and most importantly returning to the repl at the push of a button. I had to do a lot of work for this first app; get emacs working, run up against problems with the mac java implementation interacting with QT, and learn a little bit of clojure (which was the best part). The benefits are huge, though, because QT is a good platform to move forward on and because clojure is an order of magnitude more powerful than java; and I personally believe that developing from the repl is much more powerful than developing from a compile/run standpoint.


At this point, I would love to upload all files related to this. I can't, so I started a github project where I will put all the code.
Git R Dun

OK, to quickly review what you will need to get shit working:

QT - I had version 4.4.3_01
java - 1.5.0
aquamacs emacs
git,svn,cvs, and the latest versions of:
(svn from various other places)
clojure
clojure-contrib

(from joshu's git repository)
clojure-mode
swank-clojure

(cvs. I wish this project had some better regression testing systems)
slime

Watch every single presentation here:
Rich Hickey

Watch them again until you really get it.

Next up will be an emacs post; I need to remember how to use/navigate within emacs and how to integrate with slime better.