Where to from here…

Been mulling over my next project after I finish a small Rails app for my friend. I’m not sure I’ll have the time in the near future but here are the ideas that are competing in my head for… well literally… mind share.

Both use Erlang. Unraverl is the beginnings of a meta programming/parse transform library for Erlang. At this point it only implements before/after filters with method chains. Grove is a JSON Query to set comprehension library for use with Mnesia.

  • Unraverl
    1. add partial application
    2. runtime function definition from funs
    3. add pure/impure designation attribute/checking
  • Grove
    1. Make use of MochiWeb
    2. Rework the JSON query structure
    3. Move JSON Query to Set Comprehension logic out into separate module/project

Clearly Unraverl is the most flashy, and likely the most interesting bit of hacking, but whether it becomes useful later or not remains to be seen. I still think that Being able to use Mnesia from Rails or from rich clients would be really very awesome.

On top of those things I need to take the time to learn org-mode, and build at least one gen_server or gen_fsm based app.

Either way code will result!

BootStraps now with less crapyness

I’ve been getting rid of my TODO’s in my mini bootstrapping library whilst focusing on my Sinatra app and the config file looks a lot less crappy now. It even looks a tiny bit awesome. Its doesn’t do everything (still need to put my app/* directories in here) but it serves my purposes and remains flexible:

BootStraps::Initializer.configure do |config|

  #Use the vendor directory
  config.vendored = true
  config.default_env = 'production'

  #require each gem
  config.gem 'activerecord'
  config.gem 'sinatra'
  config.gem 'json'
  config.gem 'haml'
  config.gem 'twitter'
  config.gem 'nakajima-rack-flash'

  #action taken when config.db.connect is called
  config.db.connect_action do
    ActiveRecord::Base.establish_connection(
        :adapter => 'sqlite3',
        :dbfile =>  File.join(config.root, 'db', "#{config.env}.sqlite3" ))

  end

  #Sinatra settings
  config.framework.set :environment, config.env
  config.framework.set :views, File.join('app','views')
  config.framework.set :server, 'mongrel'
  config.framework.use Rack::Session::Cookie, :domain => 'localhost'
  config.framework.use Rack::Flash, :sweep => true
end

No more string literal library names (holy cow that was bad), a nice flexible block for defining the db connection code, and little method missing spice to run whatever methods on whatever framework I’m using.

Using the config has been simplified as well:

require 'config/bootstraps'

module MyMod
  class MyApp < Sinatra::Base

    configure do
      Straps.db.connect
      Straps.framework.apply_settings!(self)
    end

  end
end

Thinking about rolling it into a gem just to save myself time down the road.

BootStrapping a Sinatra app

In an effort to simplify my Sinatra app startup and configuration process I’ve built out a very simple bootstrapping library. It’s robust enough to cover the needs for a small to mid size app, but simple enough that it’s not overkill for use with Sinatra.

boostraps.rb

In my config directory I have two files: bootstraps.rb and environment.rb.

The former contains three classes DataStore, Configuration, and Initializer. When you invoke the class method boot! on Initializer it pulls in the environment.rb, loads up the configured gems and then loads up all the files in whatever library directories have been specified.

Use within your Sinatra::Base subclass takes the form:

require 'config/bootstraps'

module MyModule
  class App < Sinatra::Base

    configure do
      Config = BootStraps::Initializer.config
      Config.db.connect
      bar = Config.global[:foo]
    end
  end
end

Environment.rb

The above is nice and quiet as it should be. Behind the scenes we’ve told ol’ BootStraps a bunch of stuff about what our app looks like:

BootStraps::Initializer.configure do |config|

  #Configure the db library and settings
  config.db.lib = 'ActiveRecord::Base'
  config.db.init_method = :establish_connection
  config.db.init_args = {
    :adapter => 'sqlite3',
    :dbfile =>  "#{config.root}db/#{config.env}.sqlite3"}

  #default our environment to production
  config.default_env = 'production'

  #require a ActiveRecord on boot
  config.gems['activerecord'] = '>=2.2'

  #arbitrary setting
  config.global[:foo] = 'bar'

end

Other stuff

The best part that you don’t see here is the ability to add libraries to app/[models,ext]/ or lib/ and have them required for you. Again, the point is to limit the knowledge/effort needed to contribute new code to the app.

kinks

It’s quirky in some places (you have to specify something for a gem version even if its nil) but its a relatively clean app boostrapping solution. You can check out my current use case in more detail here so long as you ignore the rest of the app :D.

OS XI

Things I’d like to see in OS 11:

  • True micro-kernel architecture
  • 64 bit all the way down
  • MacRuby support for desktop apps (Apple should adopt this project immediately)
  • Automatic window arrangement like XMonad/Ion

That is all.

RQuery: now with | and & operators

This weekend I wanted to tackle some of rquery’s shortcomings so I added the ability to |/& your operations and also the ability to use an object in the block instead of the symbol class for the column names. I agree with the assertion that it’s bad to pollute other classes, but I still think the symbol.declaration syntax is pretty clean and there’s isn’t much likely hood of those methods ever being implemented elsewhere ( says the guy who decided to implement them :P )

Examples

The key bit is the ability to add multiple operations to the same line connected with | or & operators. In this way you can form extremely complex queries, where as before you were limited to whatever you could get out a series of anded logic. The only stipulation is that you you have to wrap the operations with parenthesis if they are on the same line, or ruby will complain about a syntax error.

#All operations need to be on the same line and in parens
#either the | operator or the & operator can be used on a singel line
User.where do |user|
  (user.age.is > 20) | (user.age.in 16,18)
end

The above example should be easy to follow but there is one important gotcha with including &’s and |’s on the same line. Because the & operator has precedence in a string of operations if you want both | and & on the same line you need to force precedence for the | where it needs to be evaluated before the & in the resulting query.

#the & takes precedence here and will be grouped with the contains "Alice" which will be
#or'd with the contains "George"
#=> name contains "George" or (name contains "Alice and age from 20 to 30)
User.where do |user|
  (user.name.contains "George") | (user.name.contains "Alice") & (user.age.from 20..30)
end

In this case the query writer wanted to get the users who’s names contained either George or Alice and then filter with an age range. What it’ll actually do is find the users who’s name contains Alice and have an age between 20 and 30, and add anyone with a name like George. To fix this you can add more parens :(

#to correct the above to the more intuitive version add parens to force precedence of the
#contains operations
#=> (name contains "George" or name contains "Alice) and age from 20 to 30
User.where do |user|
 ((user.name.contains "George") | (user.name.contains "Alice")) & (user.age.from 20..30)
end

Or you can simply move the and’d operation to the next line as the operations are evaluated top down as the ruby block is executed and they are grouped using and.

#in this sutation it would be cleaner and easier to just move the and'd statement down
#a line as all seperate lines are and'd and lines have precedence from top to bottom
#additionaly operations on seperate lines don't need parens
User.where do |user|
 (user.name.contains "George") | (user.name.contains "Alice")
 user.age.from 20..30
end

The biggest change here is simply being able to use the | instead of the default for each line (and). And finally, if you checked out RQuery before, you’ll have noticed I’m passing in an object that represents the model being queried. The advantage is there’s no class polution, and the not so obvious advantage is that it will throw and nice verbose and descriptive exception if you try and use an attribute that doesn’t exist for a given model.


#should you attempt to use and attribute that doesn't exist for a given model
#rquery will tell you before it's sent to the db
User.where do |user|
  user.ssn.is == "123-45-6789"
end
# RQuery::AttributeNotFoundError: The field 'ssn' doesn't exist for this object
#       from /Users/johnbender/Projects/rquery/lib/rquery/attrib...
#       from (irb):24
#       from /Users/johnbender/Projects/rquery/lib/rquery/active...
#       from /Users/johnbender/Projects/rquery/lib/rquery/active...
#       from /Users/johnbender/Projects/rquery/lib/rquery/active...
#       from (irb):23


My specs cover most of the code and functionality for the activerecord extension (still the only piece I’ve worked on) and you can still use the symbol syntax if you want.

#environment config
RQuery.use_symbols

#example of using symbols, you can see more at the RQuery page on my site.
User.where do
 (:name.contains "George") | (:name.contains "Alice")
 :age.from 20..30
end

You can get the gem from github as johnbender-rquery, and I won’t be updating the rubyforge one unless github fails to build my gem again ( /cross_fingers ).

Searching the abstract form in erlang

[update: I've added the head matching version of the check funs as per the recommendations in the comments. Thanks Hynek!]

I imagine there’s a much better way to do this, but in the process of writing another parse_transform hack I wanted to search for specific things within the abstract form of a module which looks something like this:

[{attribute,1,file,{"./test2.erl",1}},
 {attribute,1,module,test2},
 {attribute,2,export,[{append,2},{prepend,2}]},
 {attribute,3,compile,[]},
 {attribute,6,after_exec,{[prepend,append],to_a_atom}},
 {function,8,before_prepend_to_a_atom,2,
     [{clause,8,
          [{var,8,'S1'},{var,8,'S2'}],
          [],
          [{op,9,'++',{var,9,'S1'},{var,9,'S2'}}]}]},
 {function,11,before_append_to_a_atom,2,
     [{clause,11,
          [{var,11,'S1'},{var,11,'S2'}],
          [],
          [{match,12,
               {var,12,'Fun'},
               {'fun',12,
                   {clauses,
                       [{clause,12,...

It's a bunch of recursive lists, tuples, atoms, etc. Which means you should be able to write a simple function to dive through it and find something like an atom or number, but what if you want to match a tuple where the third item is the value "Fred"?

Here's the hack I came up with this evening:

-module(util).

-export([find/2]). 

-define(call_check, fun({call,_,{atom,_,Name},_}) -> true;
                       (_) -> false
                    end).

-define(function_check, fun({_, _, Name, _, _}) -> true;
                           (_) -> false
                        end).
find(Form, CompareFun) ->
    find(Form, CompareFun, []). 

find(Form, CompareFun, Acc) when is_tuple(Form) ->
    case CompareFun(Form) of
        true -> Acc ++ [Form];
        _ -> find(tuple_to_list(Form), CompareFun, Acc)
    end;

find(Form, CompareFun, Acc) when is_list(Form) ->
    case CompareFun(Form) of
        true -> Acc ++ Form;
        _ -> Acc ++ [Y || X <- Form, Y <- find(X, CompareFun, Acc), Y =/= []]
    end;

find(Form, CompareFun, Acc) ->
    case CompareFun(Form) of
        true -> Acc ++ [Form];
        _ -> Acc
    end.

Its fairly easy to follow how the 3 clauses of find/3 work

  • First Clause: If Form is a tuple check it with the comparison function. If the comparison function returns true add it to the accumulator and return the accumulator. If not convert it to a list and call find.
  • Second Clause: If Form is a list check it with the comparison function. If the comparison function returns true add it to the accumulator and return the accumulator. If not call find on each of the elements of the list.
  • Third Clause: If Form is anything else it can’t contain something so we simply check it with the comparison function, and add it to the accumulator if it matches otherwise return the current value of the accumulator.

The funs defined at the top as macros are the matching functions. They have to be custom built for whatever you’re looking for, and the variables you include within them must be bound in the scope where you use them.

Aside from that, there are 3 things I think this highlights: anonymous functions are a gateway drug, you only have to deal with a small set of possible data structures in Erlang, and I would love the ability to check for equality with the same syntax used for assignment/matching:

{foo, _dontcare, _alsodontcare} == Bar

In this case I ended up using the assignment, catching the badmatch error, and returning false unless it worked. This is not ideal (the try should probably match against the badmatch error at the very least), but it does work, its short and easy to read and its useful for searching complex data structures. Using find would look something like:

%% finds anything in Form that resembles {call,_,{atom,_,Name},_}
foo(Form, Name) ->
    find(Form, ?call_check).

Name fills its roll in the macro and the fun is used to check all the elements in Form. What you get back is a list of the matches. Not too shabby for 30 something lines of code!

non-deterministic description

File this under that “meta-tao of hacking” or WARNING: thar be rosy language and fluff hereabouts.

Walking my dog down the street at dusk today the air had a nice warmth to it and I enjoyed the pink and gold hues of the sunset (please stick with this post it does go somewhere). It’s warming up here, and instead of racing home with my head down I was free to enjoy the smell of the air and the sites around me. So, I was letting this scene (dusk, pink, leaves, trees) roll around in my head trying to think how I would describe it accurately to my wife so she could somehow feel and see the light as it passed through the new leaves and buds of flowers, the warmth hitting my face, etc, etc.

Mooshy

Its very hard, if not impossible to achieve that type of expressiveness. Thats what makes great writers. Transferring the totally ephemeral/sensory/emotional/mooshy stuff to these semi-deterministic buckets of words and phrases that comprise our language is a real and difficult challenge that people seek to tackle every day. Even as I’m writing this I’m struggling to lay my ideas out in a way that can be understood by someone else reading them. Without even considering the gulf of perception and personal experience that separates any two people this task is almost insurmountable.

In a lot of ways language designers are trying to do the same thing in reverse. They’re trying to give us the tools to describe a very deterministic world using words and phrases with more meaning than their underlying implementation. We’ve continually abstracted farther and farther away from the on die voltage so that a more detailed and expressive version of computational reality can be recorded.

As a result we get a meeting somewhere in the middle:

God/Universe => Words/Phrases => ---?--- < = Haskell <= Assembly <= 0011101001000100

As a programmer your brain bridges the gap there, further distilling your understanding of language and how to represent the world in written form into code so we can turn that last bit around.

God/Universe => Words/Phrases => :D => Haskell => Assembly => 0011101001000100

Sweet diagram bro

That gap from Words/Phrases to your language of choice is 6 dashes and a question mark wide. Plenty wide enough for anyone who hacks, or takes an interest therein, to feel proud. Thats the world we live in, striving to make our lives easier and our code more expressive.

As time marches on it seems like we’ve worked our way towards that reality of using languages to tell the computer exactly what to do. Slowly but surely working up from hardware, to assembler, to cobol, C, Java, Ruby/Haskell/Scala/. It’s gotten to the point now where its even efficient when it gets boiled all the way down! So what are some of the spoken language “idioms” we use every day when programming, and what are some other things that have yet to make their way into our programming?

merge! valid? Damn you GHCI compile my code!

I personally really like the Ruby way of using the language equivalent of tone to convey meaning in code. When you ask a question of an object and expect a boolean you can use a question mark (how novel!) and when I use the bang operator it reminds me of getting frustrated with clamshell packaging and just smashing my way into it with a screwdriver which is most definitely a destructive update.

Even better is Haskell’s type inference. Admittedly my knowledge of Haskell is limited to a small set of hacks that I’ve been toying with, and in almost every case I define a type signature for my functions, but inference is something we do every time we speak or write. My use of “their” or “there” in spoken form is immediately apparent because of the context its been wrapped in. Haskell’s compiler(s) is(are) smart enough to force a type based a what can be gathered about the operations over that type within a function. Thats pretty awesome.

Stuff you should ignore

There are other places that inference could be used. Namespaces/modules for instance. For better or worse, allowing the compiler/vm to handle namespace collisions could be achieved my the inferred functionality and object contained within. This of course gives people the free reign to name their modules whatever they like, thereby confusing the poor end user who needs to use two modules with completely different functionality and the same name (I think there’s already another project named RQuery).

Next Page »