Christian Sunesson

A difference between OOP and COP

2009-05-19T00:00:00-07:00

2009-05-19

Let me try to explain a difference I keep in mind for Concurrency Oriented Programming (COP) versus Object Oriented Programing (OOP). Erlang flavour.

OOP is all about where data ends up being stored. You give things that store data descriptive names that corresponds to what they keep and what they model in the real world. Threads that go around and mutate state in the objects are secondary citizens, you dont give them good names, if you try to name a thread in a given OO design it would probably be something generic, such as “Worker”, “Executor”, “Handler”. They’re things that do stuff.

In COP it is the other way around. You dont focus so much on things that store stuff. You pay all the attention on things that accomplish stuff. You want those things to do small and well-defined things. You want to give them names that describes the things they do. You want them to go away when they have done what they should.

This is “Object” versus “Process”.

Erlang packaging format

2009-05-01T00:00:00-07:00

2009-05-01

Just having read Paul Mineiro’s article The Need for Language Specific Packaging I finally see a solution for erlang software packaging that I like. I have mostly ignored all the people asking for an Erlang version of things like Perl’s CPAN. And that goes for the existing efforts, such as erlware and CEAN.

The reason I have ignored it is because I dont see myself using yet-another-tool-for-installing-software. I have been quite happy using the ones from linux distributions I know. You seem to be able to add third-party-vendor package repositories for both redhat’s dot-rpm and for debian/ubuntu dot-deb files. The problem then are “those other people” that use macs or windows or whatever.

So my take from reading Paul’s article is that what we need is a format for describing the package, so that tools can be made to automatically build package files for each package system in existance now and in the future. We do not need our own distribution system, our own package file format, our own software update system, and our own documentation.

The file could probably be something simple as a single main package description, then one file with specifics for each package file-format target. Everything in convenient file:consult/1-able text formats.

Reverse engineering oracle protocol

2009-04-28T00:00:00-07:00

2009-04-28

This is one of those “so clever it is dumb” things. Sitting with wireshark to verify system behaviour in connecting to oracle i happen to see some exchange in the protocol.

Everybody follows
Speedy bits exchange
Stars await to gl@ow"
The preceding key is copyrighted by Oracle Corporation.

The idea is likely that by having this copyrighted poem required in the protocol, no body but oracle can implement the protocol without breaching their copyright to the poem. (Not only my guess)

And those guys just bought JAVA. Yay.

Credit card usability fail

2009-04-28T00:00:00-07:00

2009-04-28

Today it happened again at the store. The guy in front of me in line forgot his credit card in the card reader, but I noticed half a minute after he had walked away into the masses. The cashier handled it and in our spontaneous discussion about it she pointed out that she gets plenty of them now when the old magnet-strips are being phased out with a chip.

Magnetic strips are slid through the reader and you stand there with your card in your hand. With chips, you insert the card, type in your code and agree with the sum to pay, then you stand there without the card in your hand. Then comes the final important step that is essential but tiny enough to forget. Take the card back.

I dont understand why they dont go the route of car seat-belts, beep, and that until you take the card out. Or flash, if the noise would be an issue.

PS

Like this idea, Banks? I would gladly accept a small symbolic fee of 0.1 cent per cc transaction. A simple token of your appreciation. Mmmkay?

Wake up again

2009-04-24T00:00:00-07:00

2009-04-24

Put on your reflective sun-glassess and get into multi-core programming in C++. Professional multi-core programming.

Erlang is for girls.

Wake up!

More erlang-gettext hack

2009-04-14T00:00:00-07:00

2009-04-14

Implemented this interface:



1> gettext_generate:compile(se, “swedish.po”).

{module,se}

2> gettext_generate:compile(es, “spanish.po”).

{module,es}

But read erlang-gettext README-generated.textile at github instead.

Erlang gettext hack

2009-04-14T00:00:00-07:00

2009-04-14

I set out to improve gettext i18n in Erlang tonight. Given the erlang constant pool being optimized so that it returns references into it rather than “consing” a duplicate copy… do you see where this is going?

My result so far is just a proof of concept, too early to be released really. I just want to spread the word and show what the concept is:

Parse gettext “.po” file
Generate erlang module that exports Mod:gettext/1, taking an atom, returning a string.
Compile said module
Use I18N:gettext('Hello World') in source, but probably as ?_('Hello World') or ?__(I18N, 'Hello World'). The existing code uses the process dictionary to store the I18N table to use, and so can I.

I’ll get back when I have better code generation. Perhaps generating AST and compiling down to a .beam directly. Right now it generates source text and doesnt even escape strings.

Oh yeah, and a link: noss/erlang-gettext @ github

Using google guice in webstart

2009-04-12T00:00:00-07:00

2009-04-12

I spent all night trying to get google guice working inside a java webstart sandbox. The results of the all-nigher to get guice working in java webstart can be read there.

In short:

FAIL

Ehh.

Processes in Erlang

2009-04-09T00:00:00-07:00

2009-04-09

Found Processes in Erlang today. Inspired me to a comment:

CO programming differs from OO programming in an important way.

A process represents something happening. If you have a process that represent a passive object, it is OO programming. In CO you use values to represent passive things. (The process identifier tends to not be a good value, since it has a beginning and end, is connected to the node that created it, wont survive node restarts, it is only good if the value represented also is short-lived.)

The process has a mission, it is doing things to reach a result. Think of the meaning of “process” outside of programming (how laws are passed in parliamentarism, how a customer support issue is handled at your company, and so on). That is what CO processes are about. Compare it to the typical OO example: A Shape superclass with Sphere, Rectangle, and Polygon (etc…) subclasses. Those classes of objects have no goal. They just exist to be used.

In many situations the difference in actual code will be quite small. There will often be one process coordinating tasks for one passive object. The difference will be in your head, in how you name things, and thus how you relate to them.

In the example of the article linked to, stocks and money changing owners, the active task is the transaction. It is a process that knows the amount of stock, the amount of money, the stock owner registry, the buyer and seller (actually, bank accounts of theirs). It has a goal to perform the transaction so that no party loses property or money, or give up.

The transaction will have to use a protocol involving the processes providing services for the passive things, a stock ownership registry and two bank accounts, to complete an atomic transaction. Using a two-phase commit, for example. The processes that provide service will likely be looked up using the values representing them. A bank might not have a process for each account all the time, but create them on demand, limiting it to only one process to exist at any time.

Of course, the pattern of performing an atomic transaction involving changes to multiple entities looked up by representing value is so common that it already exists. Mnesia.

What everybody already know about Erlang

2009-04-07T00:00:00-07:00

2009-04-07

What is this guy talking about? Lots of words with a very vague red thread, and in the end:

It has been suggested in the comments that when people say that Erlang style actors don’t share state they mean it doesn’t share memory. First, I clearly defined state in the previous article as being different from its representation. But, just as importantly, I think that saying “we don’t share memory” is a distinction without distinction. It’s an implementor’s point of view that doesn’t reflect how a user must think about actors.

Having been on the internets before, I know that when people begin to clear up (adjust?) definitions, you know something fishy is going on. I mean, shaking the definitions have been used before (what is the definition of ‘is’, anyway?). Of course, it is an argument tactics that has many success stories (prisoner of war/illegal combatant, torture/interview, theft/performance bonus, christianity/intelligent design, etc.)

The particular post above smells more like a desire to bash Erlang as it is new and different, clashing with the programmer community the author identifies with and has his experience with. Not at all for pointing out something yet unknown and/or really bad about Erlang or the actor model itself. I could be wrong. I’m waiting for the movie based on the post. He is definitely not asking for clarification on something he doesnt understand though.

For future reference, here is what everyone already know about erlang (not exhaustive, not all related to the above article):

It is possible to share state in erlang. Just create a public ets table or open a file and read/write from multiple processes. Evil shared state at work!
It is possible to have as-good-as mutable-variables in Erlang, use the process dictionary. Actually, you can even read the process dictionary of another process without it knowing. Yay for race-conditions if abused.
Yeah, mnesia. Mutable table rows. We know that too. Dirty transactions for fun and profit!
At any one moment, a running Erlang program has lots of internal state that together form the current state that will shape how the system progress from there on, and that progress will likely change lots of state isolated in processes and in the mutable storages just mentioned.
One can implement asynchronous message-passing style in your favourite language too. That is not news to us. We dont get angry if you do. We encourage you to do it. We like the programming paradigm.
Erlang distinguishes itself from other languages’ syntax. It is kind of the point.
Scalability does not come by itself from using Erlang. Software still has to be designed with it in mind.
Erlang has all-or-nothing security. And the nothing-kind is itself kind of unsecure since node-communication is not encrypted out-of-the-box.

Mnesia sucks ... not

2009-04-04T00:00:00-07:00

2009-04-04

… well no it doesn’t suck. But titles like that are good for attention. :-/

Mnesia is one of the killer applications in Erlang. A killer application that get people to take a deeper look at Erlang. Some come with the expectation that it will be a silver bullet solution to all their scalability and performance problems, or that it will be easy to move over to from their current database. When Mnesia fails to be what they thought it would be, we all get to read about how mnesia sucks.

So if you’re getting into Mnesia, my first recommendation is this mnesia overview. It is just 15 pages (pdf) and contains the rationale and a basic feature overview.

In telecommunications applications there are needs different from the features provided by traditional DBMSs. The applications we now see implemented in the Erlang language need a mixture of a broad range of features which generally are not satisfied by traditional DBMSs. Mnesia is designed with requirements like the following in mind:

Fast realtime key value lookup.

Complicated non realtime queries mainly for operation and maintenance.

Distributed data due to distributed applications.

High fault tolerance.

Dynamic re-configuration.

Complex objects.

What sets Mnesia apart from most other DBMSs is that it is designed with the typical data management problems of telecommunications applications in mind. Hence Mnesia combines many concepts found in traditional databases such as transactions and queries with concepts found in data management systems for telecommunications applications such as very fast realtime operations, configurable degree of fault tolerance (by means of replication) and the ability to reconfigure the system without stopping or suspending it. Mnesia is also interesting due to its tight coupling to the programming language Erlang, thus almost turning Erlang into a database programming language. This has many benefits, the foremost being that the impedance mismatch between data format used by the DBMS and data format used by the programming language which is being used to manipulate the data, completely disappears.

Usage scenario

For example, with pre-paid cards for mobile phones, calls progress by reserving money for the next few seconds. If there is a reserve failure, the call will be cancelled, else the seconds will pass and the reserved amount will be debited. With millions of subscribers you can easily have a hundred thousand concurrent sessions. Each session will want to reserve and debit every few second.

This problem has a real-time requirement, you cant start to lag with processing reservation requests above those few seconds, because then people will have their phone-calls dropped, and they will not be happy customers. Instead they will get another operator. And you dont want to solve your performance problems by losing customers.

Typically, when systems like these fail, you allow users to call for free rather than cancelling all calls in progress and disallowing further calls (for happy-customer reasons again). This means that downtime is quite easy to measure in amount of money lost. You just extrapolate from average traffic from the same time this day earlier weeks. Assuming calls reserve money every 10 seconds, there is 100 000 concurrent sessions and the cost per-minute is 0.69 swedish krona, this calculation tells us a minute of undebited calls is 6 430.24542 Euros (SEK:EUR exchange rate at the time this is written).

So you could buy a beefy server for each minute you’re not debiting customers. It is no longer excusable to have downtime because a comparably cheap hard drive failed, or psu burned, or computer-hall electricity surged, instead it is very excusable to have a hot standby server that can take over within a second. Relative response time to get someone to the server and fix hardware problems can easily reach an hour. Humans are slow.

This is what mnesia is built for. Not for distributed storage and querying your copy of wikipedia, or other archival purposes. Just plain online and transient data for soft realtime access. The storage limit of dets-backed tables that is at 2 (or 4?) Gb, means that 100000 users can have 20 (or 40) kilobyte data about them. But you dont want to keep that on disk, you keep it as a ram_table so you can handle the high transactions-per-second. And ets-backed ram_tables do not have any limit other than available-memory in your machine.

Self-referencing anonymous functions in Erlang

2009-04-03T00:00:00-07:00

A somewhat frequent question on erlang-questions :

Would it be a bad idea to introduce a keyword for self referencing anonymous functions?

That discussion has been had several times on this list. I think nothing is done because those that are capable of doing it don’t feel that it is important. The shell is just used for basic inspection and evaluation, not for building systems. Get distel to compile into real modules if you want more interactive development. If you want recusion in real modules you can define a named function. Giving non-trivial things a name is generally a good thing. It makes it much easier to talk about.

Anyway.

Introducing a magical keyword that refers to the surrounding closure would be a bad thing anyway. First for introducing yet another keyword. Secondly because it is only referring to one surrounding fun when erlang allow funs to return funs that return funs, and so on.

The better approach would be to add an optional name that is scoped to the fun-body. This way you can have multiple nested funs and they can refer to the outer funs or itself, and the programmer can give them meaningful names.

Erlang XML soon faster

2009-04-02T00:00:00-07:00

2009-04-02

Fresh rumour from erlang-questions OTP is working on xmlerl improvements, leaked by Kenneth Lundin at OTP himself:

We are working on significant performance improvements and new API’s for XMERL.

The new XMERL will be at least 4 times faster than the current version. The base for this is a completely new SAX parser that you also can use directly with new API’s.

The memory consumption will also be significantly reduced.

Constant-pool erlang hack

2009-04-02T00:00:00-07:00

2009-04-02

Erlang R12B-0 added a per-module memory area for constants, the literal values in a module are stored there. Before, they were allocated on the heap every time they were referenced. This meant that some kinds of optimization that avoided literal lookup-tables became irrelevant in one go (without even recompiling the source). A great example of the kind of improvements that OTP focus on: removing speed-bumps to having beautiful code.

From the release notes of R12B-0

OTP-6850: Literal lists, tuples, and binaries are no longer constructed at run-time as they used to be, but are stored in a per-module constant pool. Literals that are used more than once are stored only once.

This is not a change to the language, only in the details of its implementation. Therefore, the implications of this change is described in the Efficiency Guide.

The erlang efficiency guide on constant pools pretty much say the same thing. But Björn Gustavsson adds this very interesting remark about unloading a module and the constant pool.

The hack

If one has very assymetric access patterns to some value, Maybe millions of times more reads than updates, and this is a measured problem, one can reach for hacks such as generating a module containing the values as literals and thus have a global configuration value that will not grow your heap unecessary.

But…

As always, remember when to optimize.

Google feedfetcher feed-id

2009-03-29T00:00:00-07:00

2009-03-29

Since 2009-03-23 the user-agent header in feedfetcher requests include a feed-id part with a longer number. What is this? Can I use it to go and kill further requests against that feed?

A mystery.

The only change since the last post about google feedfetcher doesnt understand 410 gone is that now feedfetcher hits up to 12 times a day for missing feeds. :/

Erlang io:format to string

2009-03-29T00:00:00-07:00

2009-03-29

This is quite a FAQ

How do you get the result from io:format into a string.

So here goes.

There is an io_lib:format which returns the formatted result instead of printing it. Notice io_lib as module name instead of io. The result is not necessarily a flat list. No. The result might look like the following.

1> io_lib:format("foo ~p bar~n", [42]).
[102,111,111,32,"42",32,98,97,114,"\n"]
2> erlang:iolist_to_binary(v(1)).
<<"foo 42 bar\n">>
3> lists:flatten(v(1)).
"foo 42 bar\n"

But as you see, there are ways to flatten the result. It is not done per automation since all data sent to ports are flattened there anyway. Ports include files, other processes, sockets, etc. The concept is called “iolist”. If you want the byte-count of an iolist, then use erlang:iolist_size.

Finally I would like to point out a little useful function called lists:concat (which is nothing like lists:append).

4> lists:concat([foo," ", 42, " ", bar, "\n"]).
"foo 42 bar\n"

Software engineering mud

2009-03-28T00:00:00-07:00

2009-03-28

I found and read Big ball of mud (pdf, 41 pages) today. It is a paper on software architecture in the “real world”.

It has many good reflections, and remarks such as that hard-to-understand code is more evolutionary fit because it is less likely to be comprehended and changed, that is both funny and cynical at the same time.

I recommend the read.

Only green threads are green

2009-03-23T00:00:00-07:00

2009-03-23

I often see userspace threads being called Green Threads. It was a name that I had never heard before Java was released. Everyone uses it as an established name for the userspace threading concept as if it has been established for many decades. At the time of writing even Wikipedia makes it appear as if the name existed before the point Java was implemented.

Some googling and I found evidence of my suspishion. Green threads is a name FROM the Java project.

When Java 1.0 first came out on Solaris, it did not use the native Solaris library libthread.so to support threads. Instead it used runtime thread support that had been written in Java for an earlier project code-named “Green.” That thread library came to be known as “green threads.”

If anything I think the name for what Erlang has should be “Williams Processes” by what was said in this talk with Joe Armstrong.

Yeah, and so Mike came along and he wrote an emulator C. Mike is pretty good in C. So, he really, I think he and I sort of, he told me quite a lot about C and Mike had a lot of experience with concurrent languages. And so he said well, the performance of a concurrent language is predicated by three things, it is the context-switching time, it is message-passing time and it is the time to create a process. So, these are fundamental. You know, in any programming mode you start with something fundamental. And you implement that very very very very well. Then you build your tower on top of it. And so, what Mike did in the emulator there was, he broke the C and then he turned on the optimiser and he printed a heavy assembly and stated it many times and did it again and again and again.

He had insight in what was needed, started the research, and did the hard work to get there. Java can have their Green threads. Green is the color I associate with people that feel sick to their stomach. Suits fine.

Inline CSS test

2009-03-21T00:00:00-07:00

2009-03-21

Some experiments with inline style, and how feed readers filter out style-attributes or values inside style-attributes:

This should be blue.

Text here.

This paragraph has 20 pixel of vertical padding.

Text here.

This paragraph has 20 pixel of horizontal padding.

Text here.

This paragraph is in x-large font-size.

Test end.

Google Feedfetcher, meet HTTP status 410 Gone

2009-03-21T00:00:00-07:00

2009-03-21

I used to host an RSS feed on a domain I own. Someone added it to google reader or igoogle, so google feedfetcher will fetch it. But the feed was very short lived, and removed so feedfetcher got a 404 reply.

About two years later, google feedfetcher is still trying to fetch that feed, and it attempts so 8 (!) times every day.

Expecting much from google, I find the FAQ entry How do I request that Google not retrieve some or all of my site’s feeds? in their feedfetcher help. No wait… dead links to the removals page. Well, it was not really what I wanted. I want to tell it that the feed is no more so there is not sensible to fetch it anylonger.

The FAQ indicates that google feedfetcher is too good to abide by robots.txt rules to target the feedfetcher spider. You see Google is fetching it for a user, and users are more important than publishers. A publisher that ask google to stop fetching a dead feed seems to be a preposterous request.

Really google? Is this the best you can do?

Okay. Maybe it is my fault. Maybe you keep trying to download the page because 404 is so often just an indication of a temporarily broken web server. After all that is the default response for an url that doesnt exist for most of the web servers out there. You are just missing a counter to how many times or how long time you have been getting the 404. I get it. I should respond something more sensible for you to de-subscribe all users that try to get at this feed. I could have made such a mistake as well if I had implemented feedfetcher.

So looking through RFC 2616 for HTTP/1.1 I find the 410 Gone status code:

The requested resource is no longer available at the server and no forwarding address is known. This condition is expected to be considered permanent. Clients with link editing capabilities SHOULD delete references to the Request-URI after user approval. If the server does not know, or has no facility to determine, whether or not the condition is permanent, the status code 404 (Not Found) SHOULD be used instead. This response is cacheable unless indicated otherwise.

The 410 response is primarily intended to assist the task of web maintenance by notifying the recipient that the resource is intentionally unavailable and that the server owners desire that remote links to that resource be removed. Such an event is common for limited-time, promotional services and for resources belonging to individuals no longer working at the server’s site. It is not necessary to mark all permanently unavailable resources as “gone” or to keep the mark for any length of time — that is left to the discretion of the server owner.

Ah! Of course! Someone at google must have thought about this concept to let publishers kill feeds!

There is just one problem, I changed my dead rss feed to serve 410 back in october 2008, and feedfetcher is still hitting me 8 times a day…

EPIC FAIL

HTTP activity diagram

2009-03-20T00:00:00-07:00

2009-03-20

This is more of a reminder to myself for something I should use for future work in iserve. The Alan Dean HTTP Header Status diagram. That’s some beautiful work.

If there is something else to say in this post it is a complaint with myself that I am not better with tools to create such diagrams, or generally, I am not good at visualizing situations using diagrams.

Programming is so very much about communicating your interpretation of requirements and your intentions in how to solve them, and diagrams can say so much more than an A4 page full of text.

One particular school for modelling is the fundamental modeling concepts described as:

FMC is the acronym for “Fundamental Modeling Concepts”, primarily a consistent and coherent way to think and talk about dynamic systems. It enables people to communicate the concepts and structures of complex informational systems in an efficient way among the different types of stakeholders. A universal notation originating from existing standards, easy to learn and to apply, is defined to visualize the structures and to communicate in a coherent way. In contrast to most of the visualization and modeling standards of today it focuses on human comprehension of complex systems on all levels of abstraction by clearly separating conceptual structures from implementation structures.

It is a very beautiful visual language that Erlang programmers should find useful. Another thing I like with FMC is that it focuses on using the visual model for human comprehension and not for that FAIL-IDEA that is UML based code stub generation. I much rather generate visual presentation of written code than generate code from visual presentations.

Concurrent separation of concerns

2009-03-19T00:00:00-07:00

2009-03-19

I spent much of this week debugging java, from post mortem dumps, to find a problem related to that all threads in a pool were trying to talk with a remote service that had crashed. Therefore nothing else in the java process was handled, making it appear dead. It is a typical stability anti pattern to not mark the service as broken and not attempting new connections for a while. This pattern is called the circuit breaker in the book “Release It” from pragmatic programmers. (Very good book, if you’re into long-running software you should read it!).

But the circuit breaker is not what I want to get at with this post. A little trend on this blog is for me to reflect on java programming vs erlang programming.

Joe Armstrong call Erlang’s paradigm concurrency oriented programming to tell it apart from the object oriented programming that java is identified with. Also, Steve Yegge points out a difference in thinking primary about nous, or primary about verbs in his blog post.

If object oriented programming is about things that are, then concurrency oriented programming is about things that do. Nouns vs verbs.

Now, in this blog I also have a tendency to write about testable code. And there are three rules to make your code more testable (which also happen to be three rules for good architecture overall, that writing unit tests will guide you into):

Having spent time trying to debug the above java mentioned thread problem I see that in java, the programmers violates separation of concerns enormously when it comes to threads. A good rule of thumb is that if you describe somethings purpose, and use the word “and”, then you should break it up into parts until you do not use “and” anymore. In Erlang you actually do this with processes. A gen_server process is there to fullfill a single purpose (if the design is good). All the calls and casts are there to archive this one purpose. Short lived processes are spawned to do one thing and get back with the result when done.

In java, it is absolutely not so. I would not be exaggering if I said that most threads’ purpose description will use the word “and” at least 5 times. It is the thread that accepts the socket and and executes its handler and queries the database and fetches from memcache and writes to the log and talks to the credit card processor and renders the result page and sends it back.

Maybe it is just a side-effect of java programming being the kingdom of the nouns, and erlang being the kingdom of the verbs.

PS
I am well aware that one can use a more message passing approach in Java using java.util.concurrent blocking queues and copy on write arrays and hashmaps. I’m merely pointing out the “cultural” difference resulting from the programming language concepts. A point that I will try to use for future Java programming.

R13A and Schedulers

2009-03-18T00:00:00-07:00

2009-03-18

So now is the future of erlang. We have multiple schedulers. And with interesting release notes such as:

OTP-7692 The Erlang process lock implementation has been improved by Mat Hostetter at Tilera Corporation.

OTP-7852 Enhanced build environment for cross compilation to Tilera Tile architecture. Support for native ethread atomics on Tilera Tile64/TilePro
(Thanks to Tilera Corporation).

… I can only assume that the future multicore world is already here.

The erlang:system_info(system_version) output changed in R13A. It looks something like

Erlang R13A (erts-5.7) [smp:2:2] [rq:2] [async-threads:0]

Which not only fixes this erlang version issue, but also conveys some new multiple-scheduler settings.

+S Schedulers:SchedulerOnline \\
Sets the amount of scheduler threads to create and scheduler threads to set online when SMP support has been enabled. Valid range for both values are 1-1024. If the Erlang runtime system is able to determine the amount of logical processors configured and logical processors available, Schedulers will default to logical processors configured, and SchedulersOnline will default to logical processors available; otherwise, the default values will be 1. Schedulers may be omitted if :SchedulerOnline is not and vice versa. The amount of schedulers online can be changed at run time via erlang:system_flag(schedulers_online, SchedulersOnline). \\
This flag will be ignored if the emulator doesn’t have SMP support enabled (see the -smp flag).

This is what ulf wiger have to say about the system_version and I suspect the thread will continue to clear up any remaining confusion.

Erlang list comprehension with no generator

2009-03-17T00:00:00-07:00

2009-03-17

The R13A release readme has this note:

OTP-7846 There will no longer be any warnings for list comprehensions without generators, as such list comprehension have turned out to be useful.

I found it curious what kind of use or possible abuse one could have for this. So I set out for some googling and found this discussion on erlang-questions about using list comprehensions without a generator.

Thought I would share it with the blargosphere.

Erlang release R13A

2009-03-17T00:00:00-07:00

2009-03-17

Erlang R13A was just released a couple of hours ago. ~~R13A is an alpha release of R13B~~. There are some very interesting things in this release.

Correction: R13A is a beta release of the next R13B major release. We do not recommend it for use in live products. — the R13A readme

The scheduler has finer lock-granularity. Several schedulers, one for each processor/core, better process affinity and cache usage. This will mean better scalability with increasing numbers of cores, proving that Erlang is to depend on when it comes to future multi-core systems.
A beta release of wxErlang, an erlang application for making desktop guis using wxWidget
A new module called unicode for converting unicode character encodings and latin character sets.

PS.

Download the erlang torrent

Compiling erlang R13A including wx

2009-03-17T00:00:00-07:00

2009-03-17

Compiling R13A was not a breeze, wxErlang had a few dependencies for it to work. My system is ubuntu 8.04.

wxWidgets version 2.8.0 or higher!
- libwxbase2.8-dev
- libwxgtk2.8-dev
- libgl1-mesa-dev
- libglu1-mesa-dev
- libglut3-dev

Maybe these are not all required. But after installing them I could ./configure with wx enabled.

Oh, and also, it is a very good idea to actually have a c++ compiler installed. There is a package in ubuntu called c++ that fetches g++ 4.2 for you. There was actually no check for that dependency in the autoconf script, but it is documented as a requirement.

After installing erlang in /opt/erlang-R13A there is an example wx gui in /opt/erlang-R13A/lib/erlang/lib/wx-0.98/examples/minimal called minimal.erl:

%%%-------------------------------------------------------------------
%%% File    : minimal.erl
%%% Author  : Matthew Harrison <harryhuk at users.sourceforge.net>
%%% Description : Minimal example of a wxerlang application
%%%
%%% Created :  18 Sep 2008 by  Matthew Harrison <harryhuk at users.sourceforge.net>
%%%-------------------------------------------------------------------
-module(minimal).

-include_lib("wx/include/wx.hrl").

-export([start/0]).
-compile(export_all).

start() ->
    Wx = wx:new(),
    Frame = wx:batch(fun() -> create_window(Wx) end),
    wxWindow:show(Frame),
    loop(Frame),
    wx:destroy(),
    ok.

create_window(Wx) ->
    Frame = wxFrame:new(Wx, -1, "Minimal wxErlang App", [{size, {600,400}}]),

    Path = filename:dirname(code:which(?MODULE)),    
    wxFrame:setIcon(Frame,  wxIcon:new(filename:join(Path,"sample.xpm"))),

    wxFrame:createStatusBar(Frame,[]),
    wxFrame:connect(Frame, close_window),

    MenuBar  = wxMenuBar:new(),
    FileM    = wxMenu:new([]),
    HelpM    = wxMenu:new([]),

    % unlike wxwidgets the stock menu items still need text to be given, 
    % although help text does appear
    _QuitMenuItem  = wxMenu:append(FileM, ?wxID_EXIT, "&Quit"),
    % Note the keybord accelerator
    _AboutMenuItem = wxMenu:append(HelpM, ?wxID_ABOUT, "&About...\tF1"),

    wxMenu:appendSeparator(HelpM),    
    ContentsMenuItem = wxMenu:append(HelpM, ?wxID_HELP_CONTENTS, "&Contents"),
    wxMenuItem:enable(ContentsMenuItem, [{enable, false}]),

    ok = wxFrame:connect(Frame, command_menu_selected), 

    wxMenuBar:append(MenuBar, FileM, "&File"),
    wxMenuBar:append(MenuBar, HelpM, "&Help"),
    wxFrame:setMenuBar(Frame, MenuBar),

    ok = wxFrame:setStatusText(Frame, "Welcome to wxErlang!",[]),
    Frame.

loop(Frame) ->
    receive 
        #wx{event=#wxClose{}} ->
            io:format("~p Closing window ~n",[self()]),
            wxFrame:destroy(Frame),
            ok;
        #wx{id=?wxID_EXIT, event=#wxCommand{type=command_menu_selected}} ->
            wxWindow:destroy(Frame),
            ok;
        #wx{id=?wxID_ABOUT, event=#wxCommand{type=command_menu_selected}} ->
            io:format("Got about ~n", []),
            dialog(?wxID_ABOUT, Frame),
            loop(Frame);
        Msg ->
            io:format("Got ~p ~n", [Msg]),
            loop(Frame)
    after 1000 ->
        io:fwrite("."),
        loop(Frame)
    end.

dialog(?wxID_ABOUT,  Frame) ->
    Str = string:join(["Welcome to wxErlang.", 
                       "This is the minimal wxErlang sample\n",
                       "running under ",
                       wx_misc:getOsDescription(),
                       "."], 
                      ""),
    MD = wxMessageDialog:new(Frame,
                             Str,
                             [{style, ?wxOK bor ?wxICON_INFORMATION}, 
                              {caption, "About wxErlang minimal sample"}]),

    wxDialog:showModal(MD),
    wxDialog:destroy(MD).

So new gets a handle to wx that is used to construct the application. And if you’re not running the erlang vm with SMP enabled here is where it will crash with this message:

$ /opt/erlang-R13A/bin/erl -run minimal
Erlang R13A (erts-5.7) [source] [rq:1] [async-threads:0] [hipe] [kernel-poll:false]

{"init terminating in do_boot",{not_smp,[{wxe_server,start,0},{wx,new,0},{minimal,start,0},{init,start_it,1},{init,start_em,1}]}}
init terminating in do_boot ()

So run the example like this:

/opt/erlang-R13A/lib/erlang/lib/wx-0.98/examples/minimal$ /opt/erlang-R13A/bin/erl -smp -run minimal -noshell -run erlang halt

Update: Just like Mats Cronqvist mentions, I too did start over with a fresh untar after installing the dependencies needed for wx.

N ways to decrease coupling in software

2009-03-15T00:00:00-07:00

2009-03-15

Coupling sneaks up on your software project silently. Every quick-fix sums up to a mudball software design, something far from your initial intentions.

There are a few reasons why software projects grow until they stall, the costs and risks to changing the smallest detail become much too high to make financial sense. These are factors I have observed to play a part:

Bad understanding of high coupling among developers. What it does to the code base and how it makes future software redesigns expensive.
Stressed project deadlines where software is shipped that is far from ready quality-wise. Developers that dont have enough pride in their work to actually produce a good product, or are easily coerced to prematurely saying the software is done.
Problems with programming languages and tools.

These points need further explainations.

Bad understanding

If an accounting package starts out with a good separation of concerns, such as the model-viewer-controller architecture, where the view uses the external interface of the model. That means it will be easy to to add new views for the model, a desktop interface could be replaced or completed with a web based interface. But with inexperienced programmers this clear separation can start to break down, maybe the currently single view implementation, the desktop, passes classes directly to the model, for the model to populate with values. Suddenly the model has coupling with a specific user interface, additional projects do not only bear their own implementation costs, but also the changes to other components.

This failure to understand problems related to high coupling is often only gained by experience. A focus on writing automatic tests, at small unit-test level or higher automated integration tests, will display where there is coupling. In the above accounting package example, if a test needs to make sure the values are correct in the model, it will quickly be obvious that it is a bad idea to have a coupling to a large user-interface library, it makes writing the test fixture a headache. However, if it is easy to test anyway, it will also not pose a large problem if a different view would be added.

Automatic tests give direct feedback in how bad any given coupling is. Favouring low coupling.

Stressed project deadlines

Managment typically want to deliver quickly, because that means money-flow will start. This is understandable and reasonable, that is where the money comes from.
Project leaders are typically pressed between demanding managment and vague developers. They want to please everyone, but they know the direction money flows and, rightly so, takes order from managment.
Developers do what they do well, and avoid what is ill understood in the project requirements. Requirements are vague or incomplete. They form their own ideas and get to designing and writing software, because that is what they’re hired to do, not to contact the customer’s users and see what they need.

Try to get a different customer relation where features are each valued (as in money) by the customer and time-estimated by the developers. Use agile methods with fixed timeframe sprints, where the highest yield features are implemented first. Basically, deliver the smallest and most valuable feature first.

Make the customer understand that while the aim is to implement a system that is useful after each short sprint, the system might not be useful enough to actually deploy. The customer needs to take an active part and monitor what is being developed with product prototype demos. Explain that they actually get more for their money because they can make sure they get exactly what they need and not just an interpretation of a boring old requirements document, and that it will not take more time than the traditional: here-is-a-requirements-documents, see-you-in-six-months.

Implementing no more features than necessary minimizes things that can go wrong.

Tool problems

The least that is needed is a unit-test framework. All languages have them. All unit-test ought to be used with code-coverage analysis. A code-coverage report is a great way to realize behaviour in the code which has not been tested. The solution is either to test the code or remove the code. Make sure all developers know how to run the unit-test suite and read the results.

Some languages provide the features needed for low-coupling between parts of the software, but have a big difference in effort in doing the quick way and the correct way. In java it takes quite a lot more typing to create an interface and a class to implement the interface, so it can losen up coupling, instead of calling the class one wants to avoid coupling with. This can be remedied by better editors that are language-aware.

There are also conceptual “holes” with some programming languages. In OO programming the synchronous method call is a first-class concept. It is often difficult to break out of habit and implement a asynchronous handover of values between components. It is not impossible but it is simply not something one considers each time values need to be transferred between components. In Erlang, function call and messaging are both first class concepts, so developers will naturally think about what is needed on-a-case-by-case basis. Always using synchronous calls means that you get a coupling on the relative time to complete something (this particular problem always come up in RPC, each call can easily take 1ms to finish instead of 1ns).

Also conceptually, a system that produces values does not really need to know who consumes the values. A message-queue based communication channel between components means that you only need to know your queue name, but not where values go or come from. Not many software designs start out this way, it is something implemented first when need arises. For the OO world this is the Observer pattern.

Programmers that know many programming languages (and their concepts) have a larger potential for making good designs in whatever language they use.

Pi day!

2009-03-14T00:00:00-07:00

2009-03-14

Today is pi day! Because of 3.14 and today’s date.

Ok, this was silly. I’m actually just testing out multiple _posts directories for blogs, and this one is ending up in a different directory than my ordinary feed.

Userspace threading benefit is erased by performance gains over the years

2009-03-12T00:00:00-07:00

2009-03-12

Being an Erlang developer it is quite strange to see something like this about userspace threads in java.

Version 1.1 is based on green threads and won’t be covered here. Green threads are simulated threads within the VM and were used prior to going to a native OS threading model in 1.2 and beyond. Green threads may have had an advantage on Linux at one point (since you don’t have to spawn a process for each native thread), but VM technology has advanced significantly since version 1.1 and any benefit green threads had in the past is erased by the performance increases over the years.

So now we know that.

Testing Iserve

2009-03-08T00:00:00-08:00

2009-03-08

I probably broke iserve build-simplicity tonight. I’m still fumbling with common test. Writing a *_SUITE.erl is simple and straight-forward. What is difficult is to setup common_test cleanly.

And as it is appreciated that setting up iserve is a newbie delight it is probably a bad idea to make it difficult to build.

For now I am going to leave it like this. Hopefully someone can take mercy on me and help me organize the mess I have in my test/-directory.

PS.
I think I committed a dependency on emock as well. Just for the tests though.

Common Test progress

2009-03-04T00:00:00-08:00

2009-03-04

Another day, a little bit wiser than yesterday.

Apparently other people have been blogging about erlang’s common test. And he’s much more succinct than me.

Also: Copying common_test and test_server from the installed erlang, and the version suffixes are already there. I have coverage working on emock. It just worked. Temporarily I installed common_test like this (current working directory in the emock checkout topdir):

	$ mkdir etc etc/log
	$ cd etc
	$ cp -a /opt/erlang-R12B-5/lib/erlang/lib/common_test-1.3.4 .
	$ cp -a /opt/erlang-R12B-5/lib/erlang/lib/test_server-3.2.4/ .
	$ ./common_test-1.3.4/install.sh $PWD

Then run make test and the coverage reports are there. And that is how far I got tonight.

Wake up!

2009-03-03T00:00:00-08:00

2009-03-03

This is probably the funniest thing said in the defence of C++. Sadly I am not sure it was said with the amount of of irony I read into it.

Wake up, it’s 2008. std::tr1::shared_ptr<> exists, and is readily available.

Initial success with common_test!

2009-03-03T00:00:00-08:00

2009-03-03

So I read through the common test userguide again and also I noticed that scalaris uses common_test for its test suites. Equipped with some realistic testing and more insight to common_test I just threw myself into using ct for running my unit-tests.

The most essential part is that you have to install common_test. It is not difficult or anything. See my short summary:

Create a temporary dir $SOMEWHERE.
Go ahead and cp -a common_test and test_server from your erlang source dir.
mv common_test to common_test-VSN where VSN is whatever common_test/vsn.mk says. In my case it is common_test-1.3.4
Do the same process for the test_server directory (by its vsn.mk).
run ./common_test-VSN/priv/install.sh $SOMEWHERE

Common test will be storing reports and test-run logs in this directory.

The simple suite i wrote was this (common_test looks for *_SUITE to find tests):

-module(iserve_master_SUITE).
-compile(export_all).

all() ->
    [unittest].

unittest(_Config) ->
    eunit:test(iserve_master_test),
    ok.

Then I ran $SOMEWHERE/priv/bin/run_test -dir ~/src/gits/iserve/test (it is the dir where my iserve_master_SUITE.erl reside). A quick execution later and I could find a @$SOMEWHERE/index.html$ file where i could easily browse the generated report. Running the test suite several times and common_test accumulates historical test results.

It is a bit annoying that I had to do all these things to start, but in retrospect it did not hurt that much. The important thing is that it did work.

I’m not quite there with coverage reports yet, nor happy-happy-fun-fun single-command-run-all-tests setup. I feel much more likely that I will get there though.

Erlang coverage integration

2009-03-02T00:00:00-08:00

2009-03-02

So I am trying to set up a sweet automatic test setting for an Erlang codebase. The tools are in OTP already. There is cover for coverage analysis. There is eunit for running unit tests xUnit-style. There is make for compiling.

But it is not that simple to Just Use It™. I’m diving into the source for make.erl to see if there is some hidden feature to make it cover-compile my erl files for me. If it does not, then I will have to use plain old Makefiles to compile instead.

The slightly frightening application ct seem to be able to build and run test suites and generate reports. There is at least some kind of integration with cover in ct.

I just wish I could find some code that uses Common Test already.

Oh. And dialyzer should be included in this build process as well.

Lots of infrastructure setup… :/

Update: Perhaps the extensible build system and a collection of templates could be what I want.

Firefox's focus keybindings

2009-03-01T00:00:00-08:00

2009-03-01

I have a slight pain in my right shoulder from too much mousing, so I’m always inclined to to learn keybindings for those short interrupting needs to reach for the mouse.

As such, the control-l to give focus to the address bar is lovely, and if i press tab, focus moves over to the search field (or, more accurately, the google field :). That is how to google!

Moving between tabs there is alt-1 to alt-8, which will change to tab 1 through 8. Alt-9 will change to the right-most tab. This pretty much makes it annoying to have more than 9 tabs open. (Oh, and if you develop web sites, please do pay attention to the favicon/shortcut icon. When many tabs are open, the title will be truncated and the icon is the last thing displayed.)

For in-page searching there is control-f which makes a search panel appear below the page. Pressing control-f while the panel is visible will give the search field in it focus. Pressing escape will hide it. (What I miss here is the ability to find a link in the page and using only key-bindings follow it/open in a new tab.)

What I have not found so far:

A way to focus on the page itself. A single keybinding and have focus in the page so I dont have to tab the firefox focus to the page.
Moving to adjacent tabs.
Opera’s keyboard-driven link navigation (using arrow keys to move the focus between links, being able to open the focused one).

PS.

It is also quite rewarding to open up the keyboard shortcut help in google reader (press ? when the page has focus). Shift-N and shift-P goes up and down in feeds. shift-O will open the focused feed. Plain n and p will move focus between items within a feed, and o will open the focused one.

Shift-a will mark everything as read in the currently open feed. And j and k are shortcuts to n+o and p+o, that is, move to next/prev and open it.

Testable code means FP

2009-02-28T00:00:00-08:00

2009-02-28

I just noticed that there are more people that have realized that OO side-effect free programming leads to more testable code, and that it is very similar to functional programming. That is pretty much exactly what I said about how unit tests make you use functional programming practices.

Today I need to write some test-cases to demonstrate how to use emock for a realistic code-base. I need to make a convincing argument that it is a Good Thing™ to make your dependencies parametric to startup code. This is because I got this remark earlier today:

noss: I mean, I don’t want to code in a way that is enforced by my testing framework, but I want to code and have the framework adapt. (that’s only possible to a degree, of course)

Unit test aid for Erlang

2009-02-27T00:00:00-08:00

2009-02-27

I noticed erlymock today. It’s for mocking dependencies in Erlang so that tests can be performed in isolation.

I have just studied it briefly, but im not sure this is how I would want to work with mocks in erlang unit tests. I imagine a unit-test that uses something like:

  GenServerFun =
     fun (call, {do_stuff, X}) ->
       {reply, X}
     end,
  GenServerMock = mock:gen_server(GenServerFun),
  gen_server:call(GenServerMock, {do_stuff, 1}).

This would replace the full gen_server (with the separate module and all the behavior callbacks). Preferably the mock-servers should not need internal state. But it would not be difficult to add.

Testing in the small

Something is to be said about putting most of your effort into unit tests. A unit test is least fragile. They take the smallest amount of time to develop, and real unit tests should not break if anything else than the code under test is changed.

The higher you get in testing, the more likely it is that you will break tests when you change things in intended ways.

PS
Not to self, maybe I should implement that mocking library, I’ve got all weekend.

Quick and useful mocking of OTP gen behaviors

2009-02-27T00:00:00-08:00

2009-02-27

So earlier today I theorized about mocking OTP gen behaviors.

This is the resulting project emock: Erlang Mocking. There is even a little bit of documentation.

The code is so tiny that it would probably be no problem to get this added to eunit which is already in OTP now. I’ll try to get in touch with the responsible developer there once I use this a bit.

Not so useful, Dear Github

2009-02-27T00:00:00-08:00

2009-02-27

This is a little usability problem that keeps annoying me at github. The git repo address that is displayed on the project webpage at github, it has some javascript on it that change the webpage when I select it. The text turns into a textbox, to help me cut and paste it.

Apparently there was something wrong with the clone urls which made it worth to add those icons to do heavy duty select-and-paste work for me.

But… no.

It doesnt work.

That technology copies into the selection buffer. I need to mark ordinary text so i can paste it into an xterm. I use the traditional xorg xterm without silly menues. Middle button pastes from the X cut buffer.

It would work better if it just printed out the git repo address as plain text and added the href links after it. (Perhaps some margins around the text so that it is easier to aim.)

Ohh! And github would do well if it removed some javascript magic. On my slow machine I often get asked by firefox if I want to continue running the long-running scripts on github pages. Which also have the even more annoying affect that all my firefox tabs dont render while firefox is bogged down.

PS. Dont get me wrong github. I’m addicted to you. I should seriously look into paying even though you have such a good free plan.

Lessons learned in Software Testing

2009-02-26T00:00:00-08:00

2009-02-26

Today I left work early, the electric power went out at 2 PM. To have something to excuse myself for leaving early I brought a book from the company library (actually, a single shelf :). The title is Lessons learned in Software Testing.

Scanning it quickly it seemed to have some valuable insights, that is why I brought it with me. After reading a good part of it, I think that if you work with QA / Testing of software this is a must-read. Now, I’m a programmer, but I like food for thought in how to make software engineering better.

The first thing I learned from the book is that I am not a very good tester. I do optimistic testing. I only check that things when used as intended do sensible things. That is of course useful, but, testing how things work when used in different ways than intended is often more interesting.

There are 11 Chapters with small page-sized lessons:

The role of the tester: Some career advice. What arguments to take and not. What responsibilites to reach for and to avoid.
Thinking like a tester: Some self-reflection on your thinking. Makes you know what happens in your brain.
Testing Techniques: Basically good names and classification so you can talk about various methods and attack angles on testing.
Bug Advocacy: A lot about how to communicate bugs with others. How to use tools. How processes around bug reports are to be handled.
Automating Testing: This is the chapter that made me take the book with me. I’m working on an internal project to introduce more automated testing at work. Developers and QA have been brainstorming and exchanging knowledge for very many meetings now. So I have spent a lot of thought on what automated testing is good for. I think this is why that the 39 lessons on these 32 pages all made me think “AH! Why didn’t I read this book 6 months ago?”. If you’re starting a strategy for automated testing, this is a must-read.
Documenting Testing: I haven’t read this far yet. Nor anything after it.
Interacting with Programmers
Managing the Testing Project
Managing the Testing Group
Your Career in Software Testing
Planning the Testing Strategy

All in all there are 293 lessons together in the chapters, and that on 245 pages. The way the book is written, one should probably pick it up again and again to compare it to the reality you work in. Some lessons I did not understand now will probably appear much more sensible once I have experienced the situation and the problems.

Writing unit tests is reinventing functional programming in non-functional languages

2009-02-25T00:00:00-08:00

2009-02-25

So yesterday I posted this to the erlang list about testable code and functional programming. The mailing list followups disappointed me, since people just focused on the use of “academic”. Maybe that hurt some feelings.

Today I get up just to see a similar argument about testable code and functional programming at hacker news.

Hey! Someone submitted my story! This is a big day for me. :)

Repeated again:

I used to think that I liked erlang programming just because it was
quite academic, and that this was just a personal preference. But I
have come to realize that side-effect free programming is easier to
reason about. Not always easier to write though. Being easier to
reason about means that I can imagine more execution in my head than I
can in languages that rebind variables and mutate values. Because
those destructive changes defines a before-change, and after-change,
sequence, order, and there are only so many steps one can keep the
mind without the brain starting to lie to you, filling out the blanks
without telling you.

I recently read an article published in idg.se where Erik Stenman of
kreditor was quoted saying that (something like) java systems grow
complex over time in a way that Erlang does not as much. This is
something I experience with Java systems. In Java there is no cost in
adding more things that keep state. It is said that you just
encapsulate state it in your object, but that state is not as
encapsulated as you think. Problems are fixed by adding more things
with state, and more. and more. Pretty soon you have behavior that
depends on several hundred mutable variables, and the code-base feels
hard to grasp, difficult to reason back into previous code execution
to see the “big picture”. It makes architectural fixes more difficult.
And the problems accelerates, since more quick fixes are added. (I
really need to get better at describing this.)

The funny thing is that the OOP world have found one way to manage the
complexity and the code-bases that grow ugly: They are using unit
tests, and practice the art of writing testable code. (“Testable code”
is something that is simple to write unit tests for. )

You make it very easy to supply the dependencies to code under test
need. Avoid things that go out and grab values from global state.
You avoid side-effects, and the side-effects you need (database
updates, writing files etc) you make sure that you perform with a
layer of indirection so that during tests you can replace the real
object with (for example) a do-nothing mock.

What they are doing is that they are making as much of their code as
possible to be side-effect free and placing all that code in one
method so it can be called from a unit test. They are concentrating
side-effects to well-defined places, carefully avoiding mixing
side-effects and testable/test-worthy logic.

What they are doing is that they’re reinventing functional programming.

PS
Anyone care to make my argument better or shoot it down?

Software design with low coupling

2009-02-24T00:00:00-08:00

2009-02-24

The idea in dependency injection is that you should depend on an interface instead of a specific implementation. That way the implementation that implement the necessary interface can be injected, facilitating low coupling among components in your software.

Low coupling is nice for writing testable code, since you can inject your mock/dummy objects that facilitate your test-scenario. Which means you dont have to set up all components of your system for one small test. The test can run in isolation, and perhaps even concurrently with other tests.

Where do you draw the line ?

But how much should be injected? I’m asking since I actually want to know good rules of thumb. So far the only thing I have found is (actually I mailed asking Miško, and he had already written this) To new or not to new. But I find it to be a bit lacking and not matching with the actual problem situations when coding. It does define the useful adjectives “Newable” and “Injectable” though! (Read it!)

The thoughts I have had so far:

objects that you new-constructed (Newables) and stored as a member variable in your class, those objects should not escape out from the object. They are your internals and part of the logic. Other code should certainly not have dependencies into your internals.
objects passed through the constructor (Injectables) should not escape out from the object either (by escaping i mean to be returned from methods or stored in objects passed to methods), if other objects need them, they should likewise be injected to them at object construction. Feel free to return objects methods on your injectable would return to you though.

This leaves a few remaining objects, such as those that need to be constructed and returned from the method (or passed to an argument object). Lets call them “Escapable” objects. If these need injectable objects in the constructor you should resort to something like the guice provider class. A Provider knows how to construct an object, and it is injected to you. If the Escapable need both injectable and newable it might be an architectural flaw (is it possible for you to pass newable objects post-construction, using setters?).

Summary for Erlang

Each process in erlang has its own isolated memory. So naturally all your values are internal, and you can only send out snapshots from current values. So it is in a way impossible to let your internals leak out. And Erlang do provide a healthy low coupling between processes. This makes me feel that me and The Law of Demeter agree. Dont touch your friend’s private parts!

Injectables in Erlang come in the form of other processes to communicate with. Here the big no-no is to hard code names to registered processes, they should still be injected by args to spawn or gen_server init-args. The registered process is just a convenience for your supervisor to get external references when wiring you up. Your automated test wants to supply a mock process there.

If you need to construct new processes that need injectable closures, the Provider solution from java/guice is potentially a bit dangerous. The natural way would to pass a fun that returns newly spawned processes each time called. But for long running processes it is not safe to hold on to closures from other processes, they can go bad after code upgrades. Instead it would be safer to inject a process that you can ask to construct an object for you.

A plan for increased productivity

2009-02-23T00:00:00-08:00

2009-02-23

Sometimes it feels like I spend days working and get nothing done. But there is a cure for that.

The cure is to give your subconcious a chance to forget and drop issues by planning every work task in small, concrete and objectively finishable chunks.

Small, so you can finish it in a few hours at most, which helps the task to be
Concrete, so you know what work to put in, there is no vague details or planning to be done, so it is
Objectively finishable, and by that I mean that you know when you are done. No looking back to if you could have done a better job or if it was necessary at all etc.

This always get me into flow. Once you are done, you will not have a nagging feeling that you should go back and do something different. Because that is not the rules you have set for yourself. The task was finished because you had defined an objective measurement of what finished meant, and this before you started.

Sounds like scrum?

Yes it does! Scrum is all about setting up a “time-box” and concrete backlog items. With tdd you must have objective goals, because you cant write an automated test for “attractive gui” or “well written documentation”.

How I use this

Together with a few entreprenurial friends I have regular meetings where we discuss things to finish, and we each set up a “sprint backlog”. Typically we have dinner together during this get-together.

We work in 2 week sprints. So we have a sprint demo at the end, where we show each other what we have produced, and excuse ourself for what didnt make it and why we failed. This is because there should be some form of shame of not finished what you have taken on.

Each backlog entry is time-estimated, and continously updated to how many hours left there are on each task. Typically you learn that something is much easier than expected, and other things are much more difficult. I have a google spreadsheet for this if anyone is interested. Nothing fancy really.

As a guy without kids, and 2.3 training evenings per week, I manage to squeeze out 20h of backlog items while still working 40h weeks.

A different kind of sadomasochist

2009-02-22T00:00:00-08:00

2009-02-22

I just read Programmers and Sadomasochism and my first reaction was more along the lines of:

“Uh… align=right? Give that div a css-class and style it up using text-align:right in CSS.”

I guess I am a different kind of sadomasochist.

About “broken” html

Humans write HTML. The fact that browsers are lenient about mistakes is what made the web succeed (the web as based on http and html). Without it we would never have reached the break-even point where there is just too much useful things contributed to the web to decide to not publish more to it.

If the web had instead been built on a stricter markup language, lets call it SGML. Then that format would had just been a historical accident, it had quickly been replaced with a simpler alternative that works better In The Real World™.

Disclaimer

As a web developer, you should of course validate your html pages with the w3 html validator. I would avoid the XHTML doctypes and go for the HTML ones (actually, the transitional/loose one):

  <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
  "http://www.w3.org/TR/html4/strict.dtd">

  <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
  "http://www.w3.org/TR/html4/loose.dtd">

  <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Frameset//EN"
  "http://www.w3.org/TR/html4/frameset.dtd">

The advantages of using a doctype are two:

The validator will complain to you about not having one, and produce good results when you have one
It trigs something called “doctype switching” in browsers, turning them into “standards compliant mode” in how they interpret HTML and even CSS. This is especially interesting for IE and how it measures widths of boxes in relation to padding and margin.

PS

I dont much like the use of XML for data-exchange either. The format doesnt even have integers, just text required to parse to an integer. How lame is that in data exchange!?

Testable code drags OOP into Functional Programming

2009-02-21T00:00:00-08:00

2009-02-21

I think I am on to something about functional programming. Something that will make it easier to explain FP to someone with their head stuck in object-oriented programming. It takes a certain insight to test-driven design though. I’ll try to get my thoughts out on the subject in this long article.

One of my favourite blog topics is quality and testing. Software engineering produces very complex systems, it is amazing that anything works at all really. But to allow us to keep up with the ever increasing complexity demands and cost pressure, we need to look at how to improve quality both in the end product, and in the engineering work to produce it. This is something all software projects feel.

Unit tests

Google has a testing blog where Miško Hevery is brilliant in every post. I can thank Miško for much of my understanding in the needs of unit testing in Java. Of course, he writes about Java (and I do Java during day-hours) though my favourite language is Erlang. This makes me notice what applies to Erlang, and what is unecessary in Erlang. (Oh, and while I’m dropping links, the rest of my understanding of unit testing Java comes from guice. Guice tutorial)

Dysfunctional programming?

First of, the word functional in FP does not relate to functional/dysfunctional. It is a made up word using latin suffix “-al” which means something relating to, as in (giggle) anus/anal or vagina/vaginal. Basically, FP is programming related to functions. But very important: functions as they are known in math. Stone-cold side-effect free ones that map from one set of values to another set of values. I used to think it was just a bunch of mathematicians that wanted a programming paradigm name to flaunt.

To actually get unit tests written I can’t say it better myself: there are no tricks to writing tests, there are only tricks to writing testable code. A common pitfall is that your codebase is too monolithic. Everything seems to depend on each other so you can’t write a single automated test without having to rig a real SQL database and make real HTTP requests, or anything else external. This is an effect of having hard-coded dependencies that should have been parametric to the code using them.

Yay, independence day

In java the hard-coded dependencies come from using new on specific dependant classes and from static methods to get singleton objects. In erlang, they come from calling registered processes. The solution is the same in both cases. Provide the dependency as a parameter. In java this means through the constructor, in Erlang it means through the process spawn parameters (or indirectly through gen_server init args). This simple practice makes it possible to provide mocks. A mock is something that walks and talks like the actual dependency, but behaves in a way that aids in your test.

Hey, thanks for reading this far. I’m getting to the point now!

Getting to the point

Once you have split your hard dependencies, you can write unit tests on modules. And they all go something like:

  assertEquals(42, theQuestion("life?"));

Or perhaps I should use a purely functional math function as the example:

  assertEquals(3, sqrt(9));

We are writing our testable code so that we can call the side-effect free parts, parts where we have control of all inputs and can check all outputs. Writing testable java code means that you focus on making sure you can call pure functions. It just doesnt look that way with all the mocks and abstract classes that your test implements to shield off side-effects. Underneat it all it is Functional Programming. Or is it just a coincidence?

Erlang already foster you to write side-effect free code since values are immutable and variables are single assignment. Erlang is not pure though, since you can still send and receive messages from everywhere in your code. And you can write or read from ets tables anywere, or access the process dictionary. Though, interestingly. These non-functional practices are exactly the same thing that makes your code harder to test! Koinkidink again?

Another observation related to concurrency

If we look at Haskell, which is a bit more strict about side-effects than Erlang. In Haskell you can not perform side-effects without the monad. If you were not called with the Monad then you can only return a description of the side-effects you want performed to your caller, until you get to some function that has the Monad and can act on them.

Well, unless you take the monad with you wherever you go so you can always perform side-effects. That makes Haskell appear like an ugly version of visual basic. You will be sad and Haskell will be sadder.

If you stop fighting the language, you will slowly get to think of side-effects as “costly”. You rather accumulate lists of side-effects to be done. You will device problem solutions that minimizes the checkpoints in your program that need side-effects. Erlang yaws is a good example of this programming style. The dynamic page does not tell any http reply object to do anything, instead it returns what should be done to it (such as sending a redirect):

  out(_Arg) ->
      L="http://www.google.com/search?num=20&hl=en&lr=lang_en%7Clang_sv&q=yaws",
      {redirect, L}.

You will start to think of side-effects as how they must or must not happen in order. File copying must obviously read the source file before it can write to the destination file. But it is not always that apparent. The logging of a web request must not necessarily happen before the web request itself is replied to. Unless logging of successful reply is needed, but is it? That is an example of what you will think about.

Much of the speed of Erlang is from removing unecessary ordering of events. It is so trivial to start a process or ask some other process to do something for you, and then continue with that which must happen in order. This gives the user that snappy feeling.

The economy in automated testing

2009-02-19T00:00:00-08:00

2009-02-19

Before leaving work today I had a small unofficial meeting with my collegues in how we should argue for a large investment in an inhouse project to automate testing. Managment listens to economic reasoning. We developers have a hunch it will be good.

Is this something you have been through, or was it natural for everyone involved to go for automated testing?

The arguments we collected are basically along these lines:

if we write automated tests, we free up manual testing man-hours. Especially since we have different parallell product lines (stable and experimental), and we can easily test on our list of supported platforms,
with automated tests we can make more releases per year, which means we’re more agile in customer-ordered projects,
automated tests are better than buying cheap outsourced labour, so we can stay smaller and more high-competence,
finally, automated tests is a new kind of testing for us that allow us to find race conditions and deadlocks from long-running tests (plus, while writing them we will likely find new issues other testing was not as effective at finding).

It is actually quite difficult to translate the above list into absolute savings in money. It is all very vague and word of mouth that it is a good approach. In the end it is all about optimizing the profit of the product.

More-Erlang-at-Github WIN!

2009-02-18T00:00:00-08:00

2009-02-18

I just noticed that klacke joined github. Klacke is the author of yaws, and it seems like yaws is being moved over to git.

This pretty much means that all the popular erlang web servers/frameworks have moved to git now.

I really like git, because DVCS allow you to hack other libraries better. You can clone the original, and commit your fixes locally. When you think changes are good enough to share, you can make your github fork, and push your local git to your clone.

GIT makes it practical to change other open source projects, and easier to contribute.

Erlang Hype

2009-02-15T00:00:00-08:00

2009-02-15

Erlang was released as open source over 10 years ago. In 1998. It did not get much attention at the time. Inside Ericsson Erlang was developed to manage the complexity of phone switches, and meeting the up-time requirements.

What appealed to me from the start about Erlang was how it seemed to be the product of telecoms people turning software developers, and not the product of software developers turning telecom. It had a completely different attitude toward system quality.

Most programming languages (or the communities around them) have some kind of mind-share you must get into to really work with the language. When it comes to Forth it is about minimalism. For Lisp it is about language adaption. Etc.

For Erlang it is about writing systems that can operate for decades. This colors the way you think in Erlang.

In a decade, your powersupply capacitors can burn up. In a decade your hard drives fail multiple times. In a decade hardware upgrades can/will happen on the running system. You are programming in the precense of errors. You are programming EXPECTING them to happen. Because failures WILL happen if your system runs for a decade. You just dont know when and where, just that it will be costly if your system isnt working.

The past years Erlang has gained huge traction in USA. It is known as the language to make good use out of multi-core cpus. Not much else.

Please dont forget the roots of Erlang. Multi-core scalability and easy clustering are positive side effects. But Erlang is all about robust systems.

Article ideas feedback

2009-02-15T00:00:00-08:00

2009-02-15

I do not yet know what to use this blog for. It was more that it seemed so simple to start blogging using github pages (and it was!). So I just created it. :)

Some ideas I have had:

Write short snippets of code and link into the Erlang manual pages for some google linkjuice to them.
Review “Release It!” from pragmatic programmers, in the context of how the robustness patterns map to Erlang.
Write public promises of development, so I dont slack, but get stuff done.
Get a steady flow of usage examples for my projects.

What should I prioritize?

Christian Sunesson

A difference between OOP and COP

Erlang packaging format

Reverse engineering oracle protocol

Credit card usability fail

PS

Wake up again

More erlang-gettext hack

Erlang gettext hack

Using google guice in webstart

Processes in Erlang

What everybody already know about Erlang

Mnesia sucks ... not

Usage scenario

Further reading

Self-referencing anonymous functions in Erlang

Erlang XML soon faster

Constant-pool erlang hack

From the release notes of R12B-0

The hack

But…

Google feedfetcher feed-id

Erlang io:format to string

Software engineering mud

Only green threads are green

Inline CSS test

Google Feedfetcher, meet HTTP status 410 Gone

HTTP activity diagram

Concurrent separation of concerns

R13A and Schedulers

Erlang list comprehension with no generator

Erlang release R13A

Compiling erlang R13A including wx

N ways to decrease coupling in software

Bad understanding

Stressed project deadlines

Tool problems

Pi day!

Userspace threading benefit is erased by performance gains over the years

Testing Iserve

Common Test progress

Wake up!

Initial success with common_test!

Erlang coverage integration

Firefox's focus keybindings

Testable code means FP

Unit test aid for Erlang

Testing in the small

Quick and useful mocking of OTP gen behaviors

Not so useful, Dear Github

Lessons learned in Software Testing

Writing unit tests is reinventing functional programming in non-functional languages

Repeated again:

Software design with low coupling

Where do you draw the line ?

Summary for Erlang

A plan for increased productivity

Sounds like scrum?

How I use this

A different kind of sadomasochist

About “broken” html

Disclaimer

PS

Testable code drags OOP into Functional Programming

Unit tests

Dysfunctional programming?

Yay, independence day

Getting to the point

Another observation related to concurrency

The economy in automated testing

More-Erlang-at-Github WIN!

Erlang Hype

Article ideas feedback