The Idea Guy

Howto: Exceptions

2011-12-26T18:35:00.000-08:00

At work, we've been helping an intern who needs to learn Java and pass a test in a few weeks. Since we're helping him, I figure I should probably jot down some of what I've learned on my career in case it helps anyone else out.

Now, before I get started, anything I say here should be taken as something to be considered and not gospel. There are those who believe there is only the "one true way" to do things where in reality, there may be several depending on what you're doing, who you're working with/for, etc. Just look at the various coding standards on the internet for doing C coding. Oh, and at least one of them is the "one true coding standard."

I want to start with Exceptions mainly because it's the first thing I thought about writing about. Now, I'm not sure what the "correct" way to handle exceptions is, so I'm going to describe my preferred method of handling exceptions.

First, there's a big difference between errors and exceptions. Exceptions are errors that are unexpected during the normal operation of the software vs. errors which can occur as part of normal operation. For example, if you write a division function, divide by zero is an error because it can happen and should be checked for. However, if you're writing a logger and you run out of disk space, that's not something that we expect to happen, therefore it's an exception.

So, we should not rely on exceptions for any cases that we can resolve programmatically. This means there should never be the case where in the catch block we have logic that affects the results of the program. Instead, if we have any cases where we know that something can happen during the normal operation, these should be checked for without exceptions. Can't recall exactly where I found this originally, but exceptions are expensive operations when they occur when compared to checking for errors. Therefore, if you know an error will occur, it should be checked for whereas exceptions, since they should be very rare, should be caught using exceptions.

Next, I'm a firm believer in capturing exceptions as soon as possible vs. bubbling them up. The big reason for this is that now if an exception occurs, you should get as much information as you need in order to know what caused the exception. On one project, this was not the practice as some others wanted the exceptions to bubble up to a higher level before logging them. This to me is a mistake as you can't capture any of the state where the exception occurs when you just bubble it up to a higher level. However, if you capture the exception as soon as it occurs, you can log any and all necessary information.

Now, if you do have to raise exceptions up to a higher level, then I'd suggest that you first capture the exception at the lowest level, capture as much details as necessary, and then raise a new exception with all of the data stored in it. Remember, the goal is to ensure we have everything we need to figure out what happened so we can resolve it as fast as possible.

On the subject of logging, I highly recommend using some sort of logger to capture exception information. If nothing else, you have a persistent copy of the message.

That last thing I want to bring up is using the generic Exception when capturing exceptions vs. specific exceptions. I prefer to capture generic exceptions since there may be several different exceptions that can be thrown in a try/catch block, but unless there's something different that needs to bone done for a specific type of exception, I don't see the need to capture something specific. Typically, I just capture which exception occurred and perhaps the stack trace. I haven't had the need to do anything else, so it was just simpler to just capture Exception. Also, if for some reason I add a method or one changes and throws a different exception, I don't have to change my catch block.

Now, this is just what I do and it has worked very well for me so far. Now, I'm not saying that it should be done this way, but I wanted to show a way to do it and explain why I do it the way I do. Here's to hoping it helps someone.

Central better then distributed?

2011-11-05T13:48:00.000-07:00

While we all know that distributed version control systems are considered the wave of the future and the better solution for version control, there is at least one scenario where a centralized system can be, and is, the better solution.

First, what are version control systems typically used for? The simple answer is tracking changes to source code files. And source code is, typically, text. Therefore, it is easy to see what has changed and, more importantly, merge changes.

The merging of changes is the key issue here in that any version control system can merge changes between at least two versions of a file and have a single file as a result. The big benefit with distributed systems is that since the merging happens on the client, it's faster.

However, when we look at binary files, the whole thing falls apart since in most cases, a binary format cannot be easily merged. So, how do we ensure that if two people can make changes and not necessarily overwrite one set of changes with another? This is where a centralized system with locking comes into play. By being able to lock a file, and proper training, the second person would have to wait for the file to be unlocked before making changes to the file.

Now, where would this be useful? Here are a few scenarios: managing word documents for proposal, managing the images for a web site, managing diagrams for project planning and design, and managing data files that store complex data.

Granted, this isn't the most common scenario, but it's one that may exist and can be important for certain organizations or activities.

Optimization bad? No!

2011-08-25T17:10:00.000-07:00

I recently read another article that's has an anti-optimization slant to it. In this case, it's stating that students should not learn various little tricks to save a few nanoseconds off the time to execute a block of code. While this post did have a point, it still angered me.

You see, there have been a number of articles expounding that readability, and probably a few other aspects of coding, are much more important and we should listen to Knuth and not optimize! This recent article even made it sound like such optimizations aren't important anymore and that's what truly got my goat.

Let me phrase this as succinctly as possible: optimizations are at least as important if not more important than they were years ago. These optimizations were originally meant for the original computers where saving a few clock cycles made a large performance improvement. Now, with such fast processors, the theory is that we don't have to save a couple clock cycles here and there because the amount of time saved is inconsequential. Hate to say it, but it wasn't true.

You see, many pieces of software got bigger and bigger. And by getting bigger, there were many components that could be optimized. However, because each component is such a small part of the system, in theory they shouldn't be optimized. However, these optimizations add up and, in many cases, may be in a library used by many aspects of the system.

Now, let's take a step back and look at the reasons for optimization: performance, bandwidth, and memory. Most people talk about performance optimizations where we attempt to reduce the amount of time it takes to execute a specific tasks. And yes, in many cases it probably doesn't matter too much, but to state that one should not learn some of the little tricks to save a few clock cycles is very mistaken.

There are two great examples of where these little tricks are still applicable: big data and real-time processing. Systems like Hadoop process terabytes and more of data, so squeezing out every bit of performance means we get the jobs done faster and potentially with less hardware. With respect to real-time systems, a great article was recently written about the algorithms used on Wall Street and how these companies need results as fast as possible. They even eliminated firewalls in the name of performance!

Another example of where performance matters is web servers, though not in the way you may think. Currently, most web sites use interpreted languages because they're easier to set up and use. However, the use of interpreters does incur a cost in terms of performance and other factors. In some cases there's a shift towards compiled languages. Facebook has even mentioned created a compiler to compile PHP code down to executable code. (There may be a step where it's compiled to C) These savings in these cases isn't just in performance, but in power usage as the faster a request is completed, the fewer resources used. This can lead to fewer servers needed, thus reducing the energy footprint of the servers.

Memory usage is probably the one case where more optimization is desperately needed, but not done. How many application these days use up significantly more memory than their previous versions? How many apps are written in an interpreted language, including Java, and the users must take into account the virtual machine as well as the application itself? How often is memory just poorly allocated? Many Linux distros are capable of running well on a system with 1GB of RAM, but recent versions of Windows require 1GB and don't necessarily run well without more.

By optimizing memory usage, you can allow for more programs to run concurrently without as much contention. Also, you can use cheaper computers to perform the same tasks. Or, better yet, be able to keep more data in memory for those data-intensive tasks. Personally, if I could find a browser that could keep memory usage very small and still be very functional, I would be very happy as it would allow me to have my browser going in my little VM while I'm doing my development.

Part of my job is to create software that can analyze data in batches. The server that this is running on isn't all that big and has several processes running on it. The one aspect of the system that is in our favor is that it is multi-core, so running processes in parallel doesn't affect CPU performance too much, but there's still the issue of memory. So, my software is optimized to use as little memory as possible. Also, this allows the file cache to use more memory, which helps keep file system performance reasonable.

Lastly, there's bandwidth optimization. Here we look at things like compression, JSON vs. XML, and the like. This works for both for file system bandwidth and network bandwidth as reading compressed data off a disk means fewer disk seeks. Similarly, compressed data over the network uses fewer packets. The short of it is, the faster we get data to where it can be used, the better.

Now, most optimizations should be occurring at the algorithmic level. These are things like using a merge sort vs. a quick sort for a sorting multi-gigabyte file or how one distributes work over multiple cores/machines. However, the lower-level optimizations are still important. Isn't it important to save a few clock cycles per record when you're processing billions/trillions of them? Isn't it important to save a few clock cycles when data needs to be converted to information as quickly as possible? If so, then where are people supposed to learn this?

And if people keep downplaying proper optimization, how hard will it become to find developers who can do proper optimization?

Please note that I'm not saying that optimization is the most important part of software development. What I'm saying is that we shouldn't treat it as if it's the least important. Unless we know for certain that performance doesn't matter, primarily because it isn't used very often, then we should attempt make the software as efficient as possible. The process is relatively simple: make it right then make it fast. Well, efficient would be a better word since we know that it's not always about raw speed.

Ugh...getting tired. Here's to hoping this is well written enough to be understandable. I apologize if this sounds angry, but it just burns me that optimization is becoming less and less of a priority, but then we complain about computers getting slower.

Exceptions

2011-03-19T17:15:00.000-07:00

Exceptions. It's a part of programming that all of us have dealt with at one point in time or another. However, there is some disagreement as to the usefulness of exceptions or if they even should exist. Here's what I've discovered through reading a number of different articles.

First, there is a concern about speed. Some say that exceptions are slow and error codes should be used instead. However, it appears that while yes, checking an error code is faster. However, that's comparing checking a single error vs. handling a single exception. In reality, you'd probably have to check a number of errors every time a block of code is run vs. handling an exception only when it occurs. It is this that some people, in particular the designers of the D programming language, state is a true benefit of exceptions from a performance standpoint: since your normal code does not have to check for error conditions, it will run faster. Benchmarks that I read recently confirm this.

Second, exceptions make it harder to understand what happened. The issue here is that handling an exception is like a goto to a random spot somewhere in a program. If my understanding is correct, when an exception occurs, the program will unroll up the stack and display, essentially, the path from the program start to where the exception occurred. This really is a concern as most people are pretty lazy. I've worked on projects where exceptions are bubbled to the top of the program and the state of the method where the exception occurred. I've seen this numerous times and while it's useful to know where an exception occurred, it doesn't really tell you why. This is especially annoying when the code that threw the exception is in a library that's not yours.

What I like to do is if I know an exception can occur, I'll log the inputs to the method and, if necessary, any other related variables that can let me know what occurred. And I'll do this as close to the exception as possible. If this isn't done, you can't really know what happened. It's unfortunate that, at least in my experience, I'm the only one who's really done it.

Lastly, exceptions are supposed to be cases where something happened during the execution of the program that is not something that is expected. Errors are cases where something isn't correct, but is expected to occur. Divide by zero is an error in most cases as chances are, it's due to user input vs. something else in the system. A null pointer in a language like Java which doesn't expose pointers to the programmer is an exception in most cases.

The key is really is it something that causes program termination or is it something that is expected and can be handled. If it terminates the program, throw an exception. If you can and want to recover, check for the condition. Exception handlers are generally considered to be a poor method of doing something conditionally.

Back to the big issue, how do you properly capture an exception. Relying on the programmer to do it is really a poor solution as programmers are lazy and not always in a good way. I thought about this while traveling and I believe the solution is to have the language automatically log everything it can about the current state of the program to a file. It will make things easier as it will allow the programmer to know exactly what went wrong with ease.

In fact, the tool chain should do as much as possible for the programmer as it will make the programmer more efficient and also do common things that the programmer should be doing anyway.

With respect to SPL, this is the behavior that should be part of the language. If done right, the programmer will never have to worry about exceptions as it will automatically be properly handled by the language itself. The programmer can then focus only on what they can control vs. what they can't.

Data Types

2011-02-10T17:40:00.000-08:00

To continue the new language design, I'd like to talk about data types. First, I'd like to talk about the "core" data type. I think every language has it in some way as I believe they all need a way to pass arbitrary data to functions. In the case of SPL, the core data type is a bit string. Every type of data, even complex data structures, is represented as a series of bits in memory. By being able to treat all data as a series of bits in memory, we automatically have generics.

You see, by being able to treat all data as a bit string, whenever we want to pass arbitrary data around, we can do so easily since we don't need any type information. This idea came from Erlang where bit strings are part of the language. In Erlang, you can easily parse binary data into different data types, such as integers. Making this part of the core language not only makes generics easy, but working with binary data in general becomes easier.

As for the other data types, one of the goals of the language is to make things easier, which in turns should result in better software and the focus on simplicity is reflected in the data types. To this end, the number of data types should be relatively small. Emphasis on relatively. The types I currently see are these:

bit string
integer - Arbitrary-sized integer.
decimal - An arbitrary-sized/arbitrary precision numeric type. This is not a float, but an actual full-precision real number.
char - A UTF-8 character.
string - A series of chars.
boolean
array - A fixed-size collection of elements that are all the same type.
tuple/struct

The last I'm not sure about as each has it's benefits. Both are similar in that they are a set of different pieces of data grouped together, however they are different in how they do this. Tuples can have elements added/removed dynamically and do not have any type information associated with the elements, thus allowing elements of different types to be dynamically grouped together. Structs on the other hand are predefined, each element has type information associated with it, and cannot have elements added/removed dynamically. Both have advantages. I'm just not sure which is better for this language. The dynamic nature of a tuple is nice, but the ability to have a well defined, self-describing structure is also nice and fits with the principles of the language to make things safer.

Regardless, it is expected that complex data types will be created and used. For example, lists are one that would not be part of the type system, but is expected to be created as they are useful. The reason that lists are not part of the core language is because different implementations have different benefits and tying the language to one implementation didn't seem right. This is especially true since different algorithms/data structures are developed over time as systems evolve.

That's all I have tonight. I know it's not much, but I like it. While there are disadvantages to having so few types, in this case, I believe it's important as it's more important to prevent mistakes and make things as easy as possible to accomplish what you're trying to do vs. creating the fastest language.

New Job and Rails

2011-01-27T18:42:00.000-08:00

Well, I started my new job earlier this month and it's going quite well. Unfortunately, the hours suck as I have to be out of town a few days a week and it's a bit of a trip. However, this is definitely an environment where we get stuff done without a lot of overhead. And I actually get to learn new technology!

Speaking of which, I've been learning how to develop web sites using Ruby on Rails. The Ruby part is fine, however I'm not sure what I think of Rails yet. The nice parts about it is that you don't write a lot of code and it does help by generating some of the code and HTML for you. However, it's rigidity, which is supposed to make things easier, can make things quite a pain. For example, I made a mistake with the scaffolding, tried to fix it, however there's some "magic" in the background that prevented me from fixing it. I had to recreate the entire project from scratch to fix it. Needless to say, I have decided that I have to think very carefully before I have it generate any code.

Another annoyance is the templating. It's not that I don't like it, it's just that I'm a firm believer that there should be a clear separation of the code and the template. If you look at Template::Recall, that provides a very clear separation of the two. With Rails, you have code embedded in your template and, to me, that's not ideal as you still have a mix of HTML and code together. Isn't that one of the things that people complained about with PHP?

Erlang for Smart Phone Operating Systems?

2010-12-27T06:38:00.001-08:00

A thought I recently had was whether or not Erlang would be a better language for smart phones than Java. My reasoning is that smart phones need to be very reliable as they have become a primary form of communication for many people. Erlang, with a long history of use within the telecom sector, has proven that it is reliable. Also, it already has good multiprocessing, so that is one less thing that needs to be worked on for a phone.

Yes, much work would need to be done to make it work on a variety of hardware, but that's nothing new. Even then, unless I'm mistaken, Erlang pretty much is the same on every platform, so in theory we wouldn't have to install a special version of Erlang to develop apps on normal computers. We'd just have to make sure that all of the APIs exist for all platforms.

The only area that I see that would probably need a lot of work is the UI. AFAIK, there hasn't been any mobile UI work done for Erlang.

Anyway, just a thought. I personally think it's a good idea and would love to see it happen, however I think Android has become too powerful for this to happen unless somebody does this and really shows the benefits of Erlang over Java.

Android and the ParrotOS Concept

2010-12-18T17:02:00.001-08:00

I just had a revelation that I believe to be accurate: my concept of an OS that is a VM, which I dubbed ParrotOS after the Parrot VM, appears to be reality: Android. I haven't done much with it, but it does run on multiple devices, each with a different hardware configuration, and all of the software, as far as I can tell, will run on all Android devices. This is, of course, barring OS compatibility.

The coolest part is the fact that the concept works pretty well. Android phones appear to have very good performance despite the fact that the "apps" run in a VM.

Now if only we can get this to work on other devices, like desktops, laptops, and servers.

New Language Attempt Take 2

2010-12-08T13:30:00.000-08:00

Well, I decided to take another attempt at designing the language. It may not be the same, however that's O.K. I'm not 100% sure that I did things right the first time, so I figured I'd take another stab at it especially since I have some new ideas.

So, the first step I'm going to take is to clearly outline my goals. I feel that this is a crucial step so that I stay on track. If I don't have a clear picture of what I want the language to be able to do, then how do I know when I'm done or even which direction to go in?

I'll most likely revisit/reuse some of my previous ideas, however I'm sure some decisions I had made before will change. Again, that's O.K. I'd rather change my mind now to make a better product than regret an old decision.

Rust Language

2010-11-30T13:31:00.000-08:00

In case you've been living under a rock, Mozilla has been working on a new language called Rust. I've known about it for a while, though I just realized that much of what I was thinking would be good in a language is included in Rust. Here's a link to more info: https://github.com/graydon/rust/wiki

Language Ideas Part 3

2010-11-19T11:22:00.000-08:00

This part of this series is about what is one of my favorite concept in the language: iterators. This is a bit different that iterators in C++ or Java. In this case, it's really just a special type of function that automatically iterates over an array or list.

Syntactically, it looks something like this:

iterator squareAll(int[]) -> int[]
given currVal:
    currVal * currVal

It's not a finished concept. In this case, because the output is an array/list, then automatically takes the return value and adds it to the output array. Whether or not that's appropriate or not is debatable, however, it is clean. Also, it looks suspiciously like a map function. Here's a version that's like a reduce:

iterator summation(int[], init) -> int
given currVal, acc:
    acc + currVal

Here, we have an accumulator that we add each value to. However, in both cases, there is a significant flaw: we only look at one value at a time. If we need to look at previous or future values, we're out of luck. So, a recent thought I had was this:

iterator summation(int[], init) -> int
given values, index, acc:
    acc += values[i]

In this case, we now retrieve the value using the index, much like a C array. It's not the most pleasant solution, however it's still nicer than recursion. It's still not quite right as there it removes the convenience of not having to use an index to get a value. This is similar to the foreach and for loops that exist in several languages.

To go back in time a bit, the inspiration for this construct came from the D Programming Language. Specifically, the concept of ranges which have the ability to add/remove values from either end of a range as well as access values via an index. Ranges are really nothing more than some sort of data structure contained within a struct or class that has a specific API. In SPL I wanted the same thing mainly because I wanted to allow users of the language to create lists using the best data structure for their application. Each of these data structures could be accessed using an iterator without any additional work.

All of these ideas are good, however it's making them work together that's the hard part. I'm not really sure what the best solution is as I don't want to have two different constructs if one can do the job, but I also don't want to make things too complicated.

Perhaps it'll be figured out in the future.

Language Ideas Part 2

2010-11-18T06:51:00.000-08:00

State. It's not something that is particularly liked in functional programming languages as it can lead to bugs. But what are the real issues with state? To me, the real issue is state across different scopes, such as global variables. So, I did some thinking and I felt that having no mutable variables at all was too restrictive. The question was, what is the right amount of mutability?

The D Programming Language has the concept of pure functions where they cannot access anything outside of the scope of the function. However, you can modify variables inside of the function, such as a loop counter. This is a concept that I liked a lot as it allows for mutable state, but in such a way that's much safer than normal. So, I tried to incorporate something like this in SPL.

This brings us to the concept of factories in SPL. Factories are similar to classes in that they contain methods that can be executed. However, they are vastly different. First, there are two types: pure and stateful. A pure factory will always return the same result for the exact same input as there is no state stored within the factory. Essentially, it's a class with only methods and no member variables. A stateful, on the other hand, does not have that guarantee. However, the state associated with the factory is contained solely in the factory, thus preventing direct outside modification. Think of it as a class where all member variables are private.

Why is state allowed here? Simply put, there are many different algorithms that are made much simpler if state is allowed. For example, if I have a factory that implements a queue, instead of passing the entire queue from function call to function call, I can keep it within the factory and modify as needed. The key is that the mutable data is only accessible to functions within the factory and from nowhere else. This prevents any accidental modification of the data as you would have to explicitly call a method within the factory to modify the data.

This brings up the next key differences between factories and classes: message passing. The design of the language was to allow a simple interface for communicating with a factory that not only isolated the design of the factory from the code, but also could allow for performance optimizations through threading. To hit on the latter point quickly, it was envisioned that an implementation of SPL could implicitly have factories be different threads, much like Erlang processes. This can allow for asynchronous messages to be executed by the factory without the main program stopping. Think of a good logger.

Why does this make communication simpler? The main reason is loose coupling between the code sending the message and the code executing the message. My vision was that each factory would have a dispatcher that would examine a message and call the correct method. The beauty of this is that if I decide to replace one method with another, the calling code does not have to change. I can create the new method, perhaps put a new entry in the dispatcher for testing, and when it's ready, simply change the entry in the dispatcher to point to the new method. Granted, not every case is safe from changes to the caller, however the more we can make it simpler to update a factory safely, the better.

Another reason for message passing is the concept of fail-fast error handling. Erlang is designed like this and I wanted to have that concept in this as well as it's a simple and proven method to make reliable software. In short, if there is a failure in a factory, stop execution of the factory, generate error information, and send that back to the caller. The caller is then responsible for what to do if an error occurs.

In the end, I felt that this was a very reasonable language construct. It allowed for good encapsulation without being overly complex. Aspects of the system can be redesigned with minimal impact on the code using it.

Enjoy.

Language Ideas Part 1

2010-11-17T10:26:00.000-08:00

First, apologies for the long time between posts. It's been a stressful few months, but things will be getting better.

Now, before all the stress I began work designing a language to facilitate the creation of software in a very safe manner. I did this by trying to make the code as easy to read as possible, but to also prevent things that we know cause problems, such as shared global variables and lack of bounds checking. Unfortunately, two things happened: first, I lost all the work I did and second, I realized I don't have the time to really do it justice. So, I'm going to blog about it as I recall what I did and perhaps come up with more ideas that I believe are worthwhile.

The first aspect of the language, which I called SPL or Safe Programming Language, I want to discuss is the syntax of functions. What I did was I wrote out the same function in several different styles, looked at the pros and cons of each, and tried to make an informed decision about which was I thought was best. The same was done for the Zimbu language and I liked it a lot, hence why I followed suit. It worked out well as I could then see what the code would look like and I could try to catch poor decisions earlier in the process.

I ended up deciding that a functional programming style was best as it really promoted the creation of very small functions, which is a very good thing as the smaller a function is, the more understandable it is. However, I didn't like any of the current functional programming styles enough because they weren't quite as readable as I wanted it to be, so I came up with my own. Below is a sample as best as I can remember it:

function square(int, int) -> int
given x, y :
   x * y

This is designed to be read as follows: The function square takes to arguments, and int and an int, and produces an int. For pattern matching, it reads as follows: given two arguments x and y, then do x * y. You can have multiple "given" clauses to handle different cases. Here's an expanded example using guards:

function abs(float) -> float
given x when x < 0:
    x * -1
given x
    x

In this case, the first pattern is read: given a value x, when it is less than 0, then return x * -1. Again, easy to read. Throughout everything I did, I tried to not be overly dependent on symbols, but also not pollute the language with unnecessary keywords. I did tend to use keywords over symbols as they are much easier to understand in many cases.

I didn't hit a lot of possibilities yet, but I was in the early stages of the design and was somewhat ADD as I kept jumping between different aspects of the language. However, I believe this to be pretty solid and definitely fits in with the easy to read aspect of the language.

Enjoy.

One Time Pad in D

2010-07-09T18:59:00.000-07:00

I don't have the original article, but someone was bragging about how small their one time pad code was in Python. Well, I took it as a challenge, so here's my version in D:

import std.mmfile;

void main(string[] args)
{
  MmFile input = new MmFile(args[1]);
  MmFile keyfile = new MmFile(args[2]);
  MmFile output = new MmFile(args[1] ~ ".enc", MmFile.Mode.ReadWriteNew, input.length, null, 0);
  for (size_t i = 0; i < input.length; i++)
  {
      output[i] = cast(ubyte)input[i] ^ cast(ubyte)keyfile[i];
  }
}

It's rather minimalistic, but it was just an exercise to see how small I could make it. If you exclude the lines that are just braces, it's only 7 lines long. I tried to make it smaller, but I couldn't get map to work the way I wanted it to. Ah well...it's still cool.

Re-invention of basic algorithms and quick status update.

2010-06-15T14:58:00.000-07:00

First, I read the following article today and found it very interesting and I feel it's one everyone should read: You're Doing It Wrong. After reading it and on my way home from work, it hit me that this is actually a bit bigger than the writer thought it would be. What I realized was that we need to re-invent our basic algorithms periodically to work better with modern systems. This article looks at the difference taking virtual memory into account can make, but we also have multicore systems all over the place and they're not going away. Being able to take advantage of these systems requires that we adapt our algorithms to better work with them.

Please read the article to get a better understanding of what I'm talking about.

Second, I haven't worked much on the logger or ubackup of late, but I'm hoping that will change. The first issue was the way I was handling data and threads in the logger sucked. I apparently missed something in my understanding of basic threads in D and it bit me. So, I'm going to try to rework it so that it uses the new std.concurrency library that uses message passing, so that should prevent that issue from happening in the future. Part of this will be allowing a user of the library to use their own custom log writer. As for ubackup, the only change is I renamed it to ulbackup (unix-like backup). Another project on launchpad had a similar project that was named differently except on the command-line, so since they were very nice about it, I changed the name. That and it was no big deal.

Anyway, here's to hoping I make some progress on these soon.

Logger on Launchpad

2010-05-12T15:11:00.000-07:00

Just wanted to make a quick announcement that a logging library that I was working on is now on Launchpad. I have also announced it to the newsgroup for the D Programming Language. So, if all goes well, it will be included in a future version.

Link: https://launchpad.net/dlogger

NoSQL More Secure?

2010-04-20T18:01:00.000-07:00

This is just a theory, but could the current "NoSQL" solutions provide a more secure data storage solution than a relational database? Now, I don't know what type of authentication all of them have. It seems like they have little to none, however I'm approaching this from a different perspective.

SQL injection attacks are some of the most wide-spread attacks that have occurred in recent years. These attacks typically are done through a web sites input forms or perhaps a modification of the parameters in a URL.

NoSQL solutions, however, do not use SQL, therefore these attacks wouldn't work on these types of data stores. Any "queries" that are performed are most likely done in code with perhaps some filtering or ordering done in the data store and since the interfaces for these solutions vary, similar types of attacks aren't possible. Well, except perhaps if a specific data store and/or middleware was targeted.

So, I'm kind-of thinking that perhaps a side-benefit of this sort-of new movement is that SQL injection attacks or similar are much harder if impossible. I have no proof, but this is something that probably should be explored as currently I'm not 100% convinced that NoSQL is the way to go with web sites, however improved security would be a compelling reason to consider such solutions. Of course, this is assuming that the basics are covered, such as secure authentication and perhaps even link encryption.

Launchpad

2010-04-16T18:51:00.000-07:00

Well, I finally did it. I set up a Launchpad account and I currently have two projects on it. The first is just various experiments that I've been trying to work on from time to time. It's mainly there as a backup, however I figure if anything good comes from it, I can put it in it's own project and there will still be the full history, though not directly linked.

The second project is what I'm calling ubackup. It's nothing big, just a simple little backup program that leverages other software to handle the compression and data integrity. I originally wrote a version in Erlang just because I wanted to play with the concurrency a little bit. However, I realized very quickly that it wasn't the best solution as once you remove the Erlang VM, it doesn't work anymore and it bit me just before I was upgraded. I use freearc for the data compression to take a directory and compress it to a temp directory. From there, it will copy the compressed archive to a shared area.

Granted, this is not a perfect backup solution, but it's what I wanted and I figured that it would be useful for others. Now, what I did do was set it up to use a config file to handle what commands are used to perform the compression and the final copying. This way, if you have your preferred compression util and a different system for storing the compressed file elsewhere, you can do it.

The current version will be written using the D Programming Language and will be using two threads: one to compress and one to copy. This worked well with the Erlang version and I wanted to do the same in this version. Now I just have to find the time to get it working.

Windows 7

2010-04-16T18:35:00.000-07:00

Well, I finally got upgraded at work and I have to say that for the most part, it's not bad. Of course, this started out a little rough as the first thing it did while upgrading is delete every bloody file it didn't recognize. It was really disconcerting how destructive the update was. Thankfully I had all of my files backed up using Bazaar, which was a real lifesaver.

What else went bad? Well, with regards to the "improved stability" of the operating system, my laptop blue-screened twice the first day. Um...yeah. So far, it looks like a fluke as it hasn't done it again.

Now, for the nice: it looks like the OS is right where it's supposed to be performance wise...about 10 years too late. Granted, I turned off most of the effects by switching to the classic theme (E.g. Windows 95-XP), so I don't get the pretty hover effects and what not, but I prefer responsiveness over flashiness. Overall, the UI did improve for the most part, however I'm not sure how much I like the "Pin to taskbar" vs. the old taskbar where you could have quick-launch icons.

Well, what can you do. I'm just thankful that so far all I need to do my job is compatible with the new OS. Of course, most of what I've re-installed so far are PortableApps, so that makes a difference. The only thing I haven't re-installed yet is the Oracle client software and that's the only thing that I'm worried about right now.

What to do with 48 cores

2010-03-23T20:26:00.000-07:00

Multiple threads allow for various problems to be solved in different ways that were not possible in past years. For example, regular expressions may have two different paths that they can take based on the input. Normally, a regular expression engine has to try one way then back track to the other or, if I understand DFAs correctly, return to a previous state that is stored. With multiple threads, multiple paths can be tested concurrently, therefore a sufficiently complex regular expression with a sufficiently complex and large data set to match against could run more efficiently on a multi-core system compared to a single-core system, however the benefits of more cores are likely to decrease as more are added.

So, what can one do with 48 cores that will truly maximize their potential? To truly see the benefit of them, data has to be very accessible to them, hence the application should maximize RAM/cache usage and avoid disk access as much as possible. Ray tracing, if done right, would be a great application for this since it can be done in parallel very easily, however anyone interested in computer graphics would suggest this rendering it less interesting and unique. Cryptography enthusiasts could suggest password or message cracking as a potential use.

Link analysis, on the other hand, would be very interesting. Granted, the data is strewn randomly through memory, thus cache misses will most likely be common, however with a sufficient number of cores, many different paths can be taken concurrently. Specifically, I can see this being of use in intelligence gathering or similar efforts in law enforcement. For example, let's say that a suspect is found. Through link analysis, we can look at all known associates, bank accounts, etc. and try to determine patterns. A small amount of relationships may be feasible on a smaller computer, however larger organizations may have more complex relationships, especially if there is significant interaction with entities outside of the core organization. With enough cores, many relationships can be explored in parallel and the software can then infer details about the relationships, such as seeing if suspects are possibly gaining information from an insider.

The human aspect of this is only one possibility. Looking at crimes to discover patterns is another, especially at a national level certain crimes are not localized. Imagine a serial killer who travels a significant amount and has victims in many cities. Local authorities may never see this pattern or share this information. Also, such a system can look at individual aspects of each crime and infer patterns with other crimes. These relationships can then be weighted to show how likely a relationship is. Granted, this is not a silver bullet, however every bit of information that can relate different crimes together may help match up multiple clues, thus leading investigators closer to a suspect.

So, in conclusion, my idea is not the sexiest or the coolest, however I hope it is more original and potentially useful than others.

Brains-Brawn Architectural Pattern

2010-02-15T17:34:00.000-08:00

I've been wanting to do some more programming and I have an idea that I'd really love to dive into. Of course, time is a problem, but that's not why I'm posting. I'm posting because while figuring out what I want to do, I came up with an idea for a new architectural pattern for creating software: Brains-Brawn.

I tried to come up with a very formal definition for this, however I don't think I did very well, so I'm going to try to describe what I mean. In this particular pattern, I see two distinct layers: the top layer which takes care of deciding what is to be done, who does it, etc. and the bottom layer that actually does the work. E.g. the top layer is the brains and the bottom layer is the brawn.

The closest I have to a real world example is MySQL and it's derivatives. The top layer is the, for lack of a better term, database engine where the SQL is parsed, an execution plan is created, etc. The bottom layer contains all of the different storage engines that actually deal with the data.

Now, in my thinking that isn't quite what I'm thinking with this pattern. What I was more focused on was having a architecture that would allow the user to use the best programming language for the job at the appropriate layer. A higher-level language like Perl or Erlang would be used for the brains since they are more easily updated, which is good for the brains aspect of the architecture since it may have to be adapted over time. With respect to the brawn, this is where you would want to use something like C, D, or Haskell. E.g. languages that can produce efficient code. Ideally, anything at this level would be relatively dumb and optimized for speed.

A side-effect of having the brawn layer being simple is that it can be made to be very reliable. This is very nice as my plan is to have that layer updated rarely while the brains would be updated as needed. I'm not sure how well that will work in practice, but it's at least a nice thought. This is also one of the reasons why I feel a heterogeneous set of languages is a good thing: higher-level languages are typically easier to update. So if things need to change, the change can occur at this level instead of the lower level where the performance critical code is.

Now, as one can tell, the brains may not be the fastest code in the world. This, I think, is acceptable considering the advantages. Also, my expectations right now are focused on long-running processes, so any speedups to the brains are negligible. However, there are cases where I'm sure speed at the brains level is important and there isn't really anything keeping one from having a homogeneous solution, so perhaps sticking with a low-level language at both levels is acceptable at times.

Hopefully one day I'll have some real-world code to show this off and prove if it's useful or not. However, I do know one thing I do need, especially if this works: a good manager-friendly name. Brains-brawn is accurate, but I'm not sure that it's something a manger wants to hear if they're considering something like this.

Anyway, it feels good to post this. I'm a bit excited about this and hopefully I'll create something impressive that uses this.

Resume

2010-01-31T18:27:00.000-08:00

Just wanted to post that I put a link to my resume in the links portion of the page. It's in Google Documents, so anyone can save a copy in any of the supported formats.

I'll try my best to keep this up to date as my skills change.

Coding Styles and D

2010-01-17T18:09:00.000-08:00

Well, I've been working on some D code of late...nothing too interesting except I'm working on a compression algorithm. I'll get to that when I figure it out, but one thing I did was I wrote some code to do RLE compression, which is very simple and very effective on the right types of files. However, I was noticing that the code wasn't running very fast and using an insane amount of memory. For a enwik8 file that was cut in half, it took 26 seconds to encode and that was my best effort. So, I asked for help as I thought there was something seriously wrong and I didn't know if it was my fault or a problem with D in either the compiler or the standard library.

Turns out, it's both.

What was happening is that I was creating a lot of small arrays and concatenating them, which isn't the most efficient way to do things. I was under the assumption that basic operations like that would be fairly efficient since they're part of the language. I was wrong. Apparently the garbage collector isn't quite up to snuff yet, so it didn't do it's job properly. So, I followed the advice that I got when I posted to the digitalmars.D.learn newsgroup (Posting here) and I got a tremendous speedup. After I was done with the changes, I managed to get the time down to less than 2 seconds for encoding and decoding!

Now, where do coding styles come into play? Easy: I was coding as if I was developing in Java or Perl vs. a systems programming language like C. In those languages, array/stream/whatever concatenation probably would have been the norm, at least that's the way it seems in Java when I work with strings, and you get acceptable performance. In C, you can't do that so you have to do things differently, specifically you have to handle all memory allocations manually. When I switched to something more similar, my code became much more efficient.

Boy, this makes me wish I did more C-level coding at work.

Thanks again to all those who helped me out with this little problem!

Two Interesting Posts/Quotes

2010-01-07T19:40:00.000-08:00

First off, I think I figured out my new years resolution: if I find an interesting article/quote I may want to blog about, I should keep track of it so I can link to it.

Now, to discuss the two things that I can't link to right now, I recently came across two things that got me thinking. The first was how TDD is pointless. I had agreed with one point and that is that you can set your API in stone too early in development, which did make sense to me at the time. The second is a quote from someone who worked at Sun that alluded to the fact that formal practices and techniques, such as UML, may not necessarily improve the development process as in the experience of the person making the quote, what he has seen developed by people developing in their free time, with a potential lack of formalized processes/best practices, being of higher quality that those that worked in a corporate environment with formal processes.

These got me thinking a bit. First off, it hit me a couple days after the TDD article that I incorrectly agreed with the API point that was made. You see, I had worked on some code where I did write tests, however as I wrote the tests and used the functions I had written, I discovered I didn't like my original interface and changed it. So, perhaps by writing out tests beforehand, one can interact with the API and get a better feel on how well done it is?

I also write code in my free time in a more relaxed mode. I still do things in a certain way, but I don't force myself to work within a standardized process. I think this allows me to conceptualize the problem better and come up with better solutions. Also, and I think this is a key point, I think that because in a more relaxed environment people tend to code without a formal design more often, they can see how the code will really work and perhaps execute the code more thus seeing how it really behaves vs. how you think it will behave based on the design. I'm not saying this is a good practice, but perhaps spending less time on documenting a design and more time coding produces better results? Of course, I would be writing/testing/executing the code in small chunks as I can only imagine the nightmare if I wrote several hundred lines of code and then tried to run it.

Of course, I do enjoy writing small amounts of code and executing it frequently. It lets me make sure that every step behaves exactly how I want it to.

Food for thought.

Rosetta Code Submission

2009-12-11T18:14:00.001-08:00

I got bored one day, so I wrote up a couple small algorithms from the RosettaCode in D and Erlang. Added Erlang implementations of Run Length Encoding and Merge Sort. Also, I added a second version of Merge Sort for D using the 2.0 version and templates. I didn't add a D version of RLE as my version ended up being similar to the version that was already there.

Hopefully they stay for a while as I'd hate to get credit for someone else's work. The Erlang Merge Sort was fun as I have the shortest implementation on the page! Of course, it helps a lot that the Erlang standard library had both a split and a merge function. However, I did create a multi-process version. I could only try it on a dual-core machine with little improvement. I'd love to know what it could do on a larger box.

Experiment in OOP

2009-12-11T17:59:00.000-08:00

A little background: I was reading an article and they mentioned a very strict definition of Object Oriented Programming and how Erlang is really a OO language because you pass messages to "Objects". In reality, these are processes, but it got me thinking: what is the "true" way to do OOP? I read up on it in Wikipedia and two of the key principles that got me thinking were the message passing between objects and the encapsulation. So, I got to wondering, how could we do this better so that none of the implementation of the class is exposed to the user, but still get relevant data in and out while avoiding getters and setters.

Below is my little experiment written in D:

import std.stdio;
import std.conv;

struct TestMessageData
{
 string action;
 string status;
 string message;
 int value;
 float floatVal;
}

class Test
{
 TestMessageData delegate(TestMessageData)[string] dispatch;

 TestMessageData squareValue(TestMessageData msg)
 {
     msg.value = msg.value * msg.value;
     msg.status = "OK";
     return msg;
 }

 TestMessageData toFract(TestMessageData msg)
 {
     float denominator = to!(float)(msg.value);
     if (denominator != 0)
     {
         msg.floatVal = 1.0 / denominator;
         msg.status = "OK";
     }
     else
     {
         msg.status = "ERROR";
         msg.message = "Cannot divide by 0.";
     }
     return msg;
 }

 TestMessageData throwError(TestMessageData msg)
 {
     throw new Exception("Exception from throwError;");
 }

 public:

     this()
     {
         dispatch["square"] = &this.squareValue;
         dispatch["to_fract"] = &this.toFract;
         dispatch["throw_error"] = &this.throwError;
     }

     TestMessageData sendMessage(TestMessageData msg)
     {
         debug(test_new_oo)
             writeln("Received message with action: ", msg.action);

         return this.dispatch[msg.action](msg);
     }
}

void main()
{
 Test t = new Test();
 TestMessageData msg;

 msg.value = 3;
 msg.action = "square";

 auto retval = t.sendMessage(msg);
 writeln("Returned with value [", retval.value, "] and status: ", retval.status);

 msg.action = "to_fract";
 retval = t.sendMessage(msg);
 writeln("Returned with value [", retval.floatVal, "] and status: ", retval.status);

 TestMessageData msg2;
 msg2.value = 0;
 msg2.action = "to_fract";
 retval = t.sendMessage(msg2);
 writeln("Returned with value [", retval.floatVal, "] and status: ", retval.status);

 try
 {
     msg2.action = "throw_error";
     retval = t.sendMessage(msg2);
 }
 catch(Exception e)
 {
     writeln("Caught: ", e.msg);
 }
}

It's not the prettiest, but pretty wasn't a goal. What I was trying out was to send messages in a standardized format, for the class at least, and to hide the implementation details as much as possible. The messages are structs that contain enough data to handle the data we want the object to work upon, what we want the object to do, and the "status" of the result. The status may be the least important, but I put it in as there may be times we get an error that's not considered an exception, therefore we don't want to throw an exception.

As one can see, I performed several operations to show different ways this could work, including errors that do/don't throw exceptions. What I really like about this is that there's only one way to interact with the object, but you still retain the ability to perform a number of operations.

Now, on the problem side, my variable naming sucks, but that's besides the point. While this seems very clean an elegant to me, especially if we want to change out implementations of methods, which in this case means changing a delegate (function pointer) from the old to the new, I'm slightly concerned about performance. The struct takes up some space as well as the dispatch table. Also, is there added overhead from using the dispatch table?

I really don't know. I haven't made up my mind 100% yet, but I felt it should be presented to the world as I think it's at least an interesting exercise. I'm sure there are ways to mitigate some of my concerns. I do have a feeling, though I haven't confirmed it yet, that there are advantages to this type of development. Time to continue pondering...

Functional C

2009-11-30T18:55:00.000-08:00

I've been toying around with the idea of a new language and the functional paradigm is definitely where I want to focus my efforts, but the question is, what would make it stand out from the rest of the field? I haven't looked at all of them, but my impression is they typically are oriented to more application-style programming. In other words, they focus on writing applications vs. something more low-level, like operating systems and embedded systems.

So, what I'm thinking is that we need a good functional programming language that can be used for systems development. This is a bit iffy as one of the key features of C, for example, is that it has very predictable behavior. Functional languages, however, don't have this as they typically have garbage collection. Also, Haskell has laziness, which isn't very predictable as well.

Can this be done? I think so, but it has to be done carefully. Without garbage collection, I'm not really sure how to best do this. Granted, a very smart compiler could translate any recursive functions into loops, but I'm not sure if that's enough. It may be as there's nothing that says you couldn't have loops in functions where mutable variables are localized and can only be used within the loop.

Now, what would the benefits of this be? Well, the fact that functions will, by default, be pure allows programs written in this hypothetical language to be run concurrently without worrying about concurrency issues as much as with C. This will be nice in applications which inspired me to think about this language: high performance computing. A recent article stated that using Haskell instead of C for one system resulted in a significant performance increase. My guess is that this is due to better concurrency.

One may think that this means that Haskell is good enough, but what about embedded systems that require more predictability? I'm guessing that robotics falls into this category. Random garbage collection could prove interesting. In fact, I recall someone stating that they built a robot or something similar that had software written in Java. If true, it garbage collected itself into a wall.

Now, some other things that I think should be in the language before I sign off:

Support for all of the different data types as C so that we can perform the same kind of bit manipulation as we can easily do in C.
Arbitrary precision arithmetic. Large number crunching and ensuring that the results are accurate, especially for financial systems, are common enough that I don't see why these shouldn't be part of any new language.
Good string processing. Processing data is a very important reason to use computers, so let's make it easy.
Fail-fast capabilities. This is one of the great features of Erlang and it should be incorporated if possible. How this will work will be an issue because I don't want this to run on a VM, which is how Erlang works.
Good threading with message passing/STM/something. Perhaps a combination of one or more as they both have their benefits.

That's all for now. I'm sure I've forgotten something I thought of, but I wanted to get this out. I think it's an important idea because if this can be done, I think it'll be a great boon for proponents of functional programming as it now can be used where C has been the best if not the only choice for developing systems. As for the name of the post, well, that's my unofficial code-name for this particular language as it encompasses what I believe it needs to be. Of course, this also led me to think of something else: can we program in C in a functional manner? Perhaps all this language has to be is a high-level language that gets translated into C written in a functional manner? Time will tell.

Last post and more...

2009-11-08T17:29:00.000-08:00

Well, I got through the problem I was having with Haskell. It was just a little type mismatch, so it ended up being no biggie. I think the tutorial just confused me a bit.

Now, while doing some work around the house, I was doing some thinking and I don't know if anyone has thought of this yet, but I think we can leverage multi-core processors to enhance single-threaded apps. Now, don't take anything I say as gospel as I'm not a chip designer; I'm just throwing out some ideas that at least I thought were pretty cool.

First, instead of branch prediction, could we instead have multiple cores pre-loaded with the instructions for each branch and just switch to the correct core depending on the result of the branching? In fact, if the processor/compiler is smart enough, it could keep those branches in cores for an extended period of time and just switch between them. This could make processors simpler as you wouldn't have to worry about good branch prediction algorithms and how much space they take up on a CPU.

Second, keeping along those lines, how about pre-loading the bytecode for different functions on different cores? Now when I call a function, I don't have to worry about loading the function every time I call it; it's already in a single core's cache and ready to go.

Lastly, should we look into making multi-core processors more interesting? I'm thinking of mixing one or more specialized cores with multiple general purpose cores. For example, a useful core would be one that can do arbitrary precision math without relying on a normal floating-point unit. Or perhaps a crypto core which is optimized for the various operations that cryptographic algorithms use? Now even if our software isn't threaded/forked, we can leverage multi-core capabilities.

Now, I'm thinking that if the first and second ideas are good that we'd want to have instruction caches on a per-core level and a shared data cache. Also, I'm thinking that many simple cores will work better than few complex cores, which is where we are today with Intel and AMD processors. Regardless, the third idea could be incorporated with either type of multi-core processor.

Thoughts?

Haskell Revisited

2009-11-07T08:51:00.000-08:00

Yeah, I'm back to looking at it and it really wasn't by choice. You see, I created a small program using D to launch scripts based on some parameters specified in a config file. We need this at work to be able to run some scripts on Windows and Linux. Instead of having to maintain two sets of scripts and batch files are very limited in what they can do, so using a small program seemed like the best way to go. It worked very well except it was compiled against a different version of glibc than what was on the target system. I guess that's not too bad considering it was the only bug found :-)

So, why the visit back to Haskell land? Well, outside of LDC, there's no real 64-bit compiler. This confuses me a bit since 64-bit systems are becoming more popular, so why wouldn't Digital Mars have an official 64-bit compiler? Yeah, got me. So, I figured I'd switch to a language that does have 64-bit support, but something newer that has facilities to write safer and better programs without needing a VM. Haskell seemed to be a good choice, so figured it would be a good choice. Also, I was listening to some presentations related to Erlang, Haskell, and functional programming, so I guess I drank some of the functional programming kool-aid :-)

Well, so far I'm hating it. Perhaps I'm just not in the right frame of mind, but it seems to be a pain so far to get command-line parameters working. I'm just following a tutorial with some changes to make it work the way I want to, however I'm getting a type error that's baffling me. Perhaps I just need to read the documentation some more, but it's frustrating that something that was very straight-forward in D isn't as nice to implement in Haskell.

Perhaps I'm jumping the gun, but command-line parameters should be easy to do in any language now. It's been done for years and, hell, just steal the GetOpt::Long from Perl like D did and be done with it. It works very well and is easy to use. However, I will say this much: Haskell at least supports the correct Unix/Posix style parameter format.

On a slightly different note, after listening to a presentation on Haskell, I'm wondering if perhaps D, which does have the concept of purity if you explicitly enable it for a function, should have functions be pure by default and explicitly make them impure. It's more consistent with their new thread local storage model and in reality, it's probably best to have the compiler assume and verify that functions are pure unless the programmer explicitly states otherwise. The big problem I see is that I/O becomes more difficult.

Open Source Voting

2009-10-24T07:21:00.001-07:00

I'm a little behind on things I want to blog about, but I wanted to get this post out quickly. It looks like there's more interest in Open Source Voting now. This is a great step forward in having the voting system be completely transparent.

The system uses Ruby on Rails for the software. While this wouldn't have been my choice, it's still a good choice due to it allow the developers to quickly implement the software and if any bugs come up, they can quickly fix them. Personally, I think I would have used a language better suited for reliable programming, like Erlang or, depending on how the UI is implemented, Haskell or the D Programming Language. This isn't to say that you couldn't make very reliable software with Ruby because people have done it successfully with C on numerous occasions. These languages just provide more built-in features to ensure reliability.

All in all, a good step forward.

Long line lengths suck

2009-10-12T18:41:00.001-07:00

I hate to vent, but I feel the need to. You see, in late August/early September on the one project I'm working on, some of the people on the project decided to revisit our coding standards. The only topic I argued was line lengths. I'm personally a fan of the 80 column rule mainly because it allows me to view two files at the same time without any horizontal scrolling. My work monitor supports a resolution of over 1600 pixels wide. Nice :-)

Well, I was overruled because, to paraphrase, it's too much wasted space and that if we are to do 80 columns, we may as well do only 10. It's so terrible to scroll vertically, that we must use up all of the horizontal space possible!

Perhaps it's a difference between those of us who have at least been exposed to real programming and Java code monkeys. (Btw: I like the guys on the project, but that doesn't mean I like all of the decisions I'm ignored on.) Well, it has bitten me in the ass recently and I'm guessing it won't change.

Now, if I do any work in Eclipse, we have an automatic code formatter that we must use to reformat our code when we save. Well, the tool followed the rules it was given and took the nice statement that I reformatted to fit on the screen and made it god-awful long. First, I believe that manual line-length formatting should be preserved, but that's not the real issue. The real issue is that I can't see the whole line of code. I have a wider monitor that most of the people on the project and I CAN'T SEE THE WHOLE LINE OF CODE! WTF?!?!?! This is supposed to be a good thing?

In my opinion, it's completely moronic. I understand wanting to allow longer lines, like perhaps 100-150 characters or whatever will fit on a common screen where I work. But I have no idea how long this line is and I have no idea what the limit it. Lesson to learn here: if you're going against convention, make sure it fits on the damn screen.

Again, apologies for the rant, but I had to vent.

Fossil is Great!

2009-09-27T18:18:00.000-07:00

Not sure how many of you have heard of this tool, but there's an SCM tool called Fossil that I have to say is really cool. I won't go deep into details, you can see them on their site, but in short, it's a single-executable that's a SCM tool, wiki, and bug tracker all in one. Yeah, that's pretty cool.

I haven't delved too far into it yet, however I did discover something that made it very usable for me: I can use the wiki to record any ideas I have and store them with my source tree. I did this not too long ago where I wanted to record an idea of mine, but didn't really know where or how to do it. By using the Fossil wiki, unless I'm mistaken, I have full access to this wherever I am. I know for certain I can run Fossil as a service and connect via a browser, but I'm also pretty sure I can pull from the repo and start it locally, make changes to the wiki, then push the changes back to the main repository.

This and the fact that it runs in Linux and Windows, I can use it on just about any computer I come across without much trouble. Even beyond the single executable aspect of it, it's stupidly simple to set up the wiki.

For anyone who needs some simple version control and more, I highly suggest looking into this tool.

Unix Utils Update

2009-08-11T19:33:00.000-07:00

A thought hit me today: wouldn't it be nice to upgrade all of the basic Unix utilities? Specifically, rewrite them in the D Programming Language. It's not a necessity by any stretch of the imagination, however it may be nice to use what may very well be a better language and create what could be a more stable, reliable, and safe.

A lot of those utilities are pretty old and things have changed over time. Computers have become more powerful and we've discovered better programming techniques. D has built-in DBC and unit testing as well as several other aspects of the language that help you make more reliable software. Even if you just think about what good unit tests could do to improve software quality, that's a nice upgrade. In fact, think about a distro like Gentoo where you install it from source. You could potentially have it unit test your code before having it compile the actual executables.

Again, this is not something that needs to be done, especially since the 2.0 version of the language is still in an Alpha state and the utils still work. It's just something to consider for the future.

D as a Learning Language?

2009-08-09T19:09:00.001-07:00

I was having a thought recently that the D Programming Language may be a good choice as a learning language. No, not as an intro language, but one where you can teach advanced data structures and good programming practices. The second is the key part since D does have things built into the language, such as unit testing, documentation generation, and design by contract, that it can be a one stop shop for professors who want to introduce their students to these practices. That is, of course, assuming that these concepts are taught as a collegiate level. I know in my experience, they weren't, but I'm not 100% certain that they were as popular back then as they are now.

My two favorite practices are the combination of unit testing and code coverage so that you can test your code and get an idea of how well you're testing it, and the documentation generation as now a student can see how their code comments can become useful documentation without much additional effort.

Language choice a micro-optimization?

2009-07-26T13:09:00.000-07:00

Last weekend I read an article, whose link I wish I kept, that choosing the language for speed reasons is a micro-optimization. If I recall correct, the article led one to believe that the time a developer spends to create the applications is more important than the speed of the application itself. Granted, there are situations where this is true, however I have to disagree with the article because I don't believe it's a micro-optimization, but a macro-optimization.

First, there are applications where speed is of the utmost importance. Let's take a look at a common one: networking. Let's say you have two choices: Python and C. Which would you rather use to write the code to send/receive messages over a network? Personally, I'd use C. Overhead can be a killer, especially if you consider that there are applications, like web servers and databases, where you have to handle a large number, potentially thousands, of requests per second. Wouldn't you want your networking code to handle all of these requests as efficiently as possible? With Python, you have some overhead by virtue of it being a dynamic language with a garbage collector and, for lack of a better term, enhanced data types that make it easier to write programs. All of this imposes overhead on the runtime of the program. Let's say it's only 30%. Would you want to lose 30% of your network performance just to save a few days of development time?

Second, there's more to performance than just computational speed. What about memory usage? For those who have followed this blog, you'll know I reduced the memory usage of a program I wrote by switching from Perl to D. For the dataset I was testing with, this means I went from 600MB of memory to only 60MB. That's a great nicety when you have multiple applications running on a system.

To me, these are not micro-optimizations, they're macro. It makes me wonder how many applications have been written under the other authors assumption would be better if the person or persons who wrote the software took the time to consider the impact of the language on the runtime performance of the app. I know one that I used would crash every time I tried to load a second large data set in it after I loaded, examined, then closed a large data set. It was written in Java and it would not free the memory it already used, even though it was not using that data anymore. I'm not 100% sure it was because of the Java runtime, but I do know that Perl doesn't free any memory it allocated until the program ends.

Parallel Regular Expression Engine

2009-07-13T17:52:00.000-07:00

Here's an interesting use for multi-core/processor programming: parallel regular expression engines. Now, granted, this only works on certain regular expressions, but in the cases where the regular expression has to choose between two or more different paths, it can slow down a regex, especially if it's complicated. Now, if you do each branch in parallel, then as paths don't match, they simply stop processing. At least this is how I see it working as I'm thinking in terms of Erlang-style concurrency. From the perspective of a programmer, I don't care about the paths that fail, I only care about the path that succeeds.

Now, I'm not 100% sure how much of an improvement this will give us in the normal cases, however for more complicated patterns may see some significant benefit, perhaps even making them more possible. And by more possible, I mean make them more likely to run in a reasonable amount of time.

We need MySQL/Drizzle

2009-07-07T15:02:00.000-07:00

Yes, I said it. We need them. I came to this realization yesterday when I saw an article about TokuDB and how it apparently can save you money by being energy efficient. The premise is simple: fewer disks seeks = less energy used.

This type of development is just cool and I don't see it with any other product. This not to say that PostgreSQL and Firebird aren't cool. MySQL and Drizzle just provide opportunities to do some very cool things due to the fact they're both very modular. If we didn't have them, I think the DB world would be more boring. I just hope we can get to the state where we can have a lot of the nice features that a PostgreSQL or Oracle have and still have the flexibility of a MySQL or Drizzle.

Functional/Pure OO Programming

2009-06-21T09:08:00.000-07:00

I saw an article about this a while ago and while somewhat interesting, I didn't really put too much stock into it as that wasn't how we were doing things on my one project as work that uses Java. Yesterday, I saw this on stackoverflow: http://stackoverflow.com/questions/1008803/how-to-use-pure-in-d-2-0

Needless to say, it got me thinking about this more. I guess it's the difference between something written up as theory vs. a real-world solution to a problem. Also, in my mind, having an example in a language that actually cares about purity reinforces this.

Right now I'm contemplating the best way to do this based on the examples that were provided in DK's reply. I'm leaning towards the second for a couple reasons.

I think it's cleaner to do all the data manipulation before you create the new object, not after.
The second solution can provide better encapsulation. You see, in the first example, it's assumed you can manipulate the variables in the class directly, which could lead to other types of errors if one is not careful. By manipulating the data strictly within a method and passing the new data into the constructor, I believe it will be safer and more reliable.

Granted, I do see the concern that without a really good garbage collector, you are create a lot of "garbage" when you do this, but I guess you have to weigh that against the benefits you get by having code that's safer to run in a multi-threaded environment.

Of course, if you're doing this in C++, memory management could be an issue if you're not careful. On the other hand, if you know what you're doing the code could still be quite efficient. In my opinion, the best solution here is to put the results into a temp variable, delete the original variable, and then assign the new class instance to the original variable. Not exactly the most efficient thing to do when you think about it. Perhaps we could see some better efficiency if we use a pool of objects and periodically clear out unused instances when you hit the end of the pool?

Food for thought.

D vs. Perl Development

2009-06-03T19:20:00.000-07:00

This really isn't an analysis of the two languages. This is more an analysis of my efforts rewriting a program in D that was originally created in Perl.

So, what's the verdict then? It's taking longer, but not for the reasons you may think. Part of it is obvious: I've had to write code that already existed on CPAN and I'm having to do it again, but that's not the interesting part. The main reason it's taking longer is because I'm writing unit tests to check as much code as possible. I didn't do it with the Perl version since it was written quickly, but now I'm forcing myself to do it. It's paying off, but it's killing my dev time. As much as I know I should keep following this pattern, I'm just itching to get this done.

The good news: I have some proof that my code is rock-solid :-)

No need for bare metal speed?

2009-05-24T14:34:00.000-07:00

Yesterday I saw a comment in a performance thread where someone stated that we don't need the "bare metal" speed we get with languages like C, C++, and the D programming language. I personally disagree. How many applications do you use that aren't written in some variation of C or C++? Probably not a lot. Why? Well, wouldn't you want the best possible performance when you're compressing files or working with large images?

There are two factors with performance: actual computational speed (processor-dependent) and resource usage. This person may have been focused on the first and is accurate to a degree, however, would you want to use a spreadsheet written in Python? Perhaps a browser written in Ruby? I'm guessing the answer is no, you don't. I'm sure there are power users of spreadsheets that already complain about performance, so why use a slower language? As for browsers, how many of you already complain that some sites are too slow? Imagine if your browser slowed them down even more? Personally, I'd rather have the software take full advantage of the CPU to make the experience as seamless as possible.

As for the second factor, resource usage, if you've followed several of my recent posts, you'll know exactly what I'm talking about. When you can reduce the amount of memory you use by a factor of 10, that's a very good thing. This is, I think, the more important aspect of those three languages vs. many others. Imagine if your office software used more memory...let's say 3 times more memory. On my current linux system, OpenOffice.org currently take up about 20MB of memory with a small document open. Multiply by three, now that's 60MB. Granted, thats only a small document. Imagine if you're talking MS Office with Outlook open plus several larger documents (50 pages). That's going to put a nice hurt on your memory usage.

Now, so far I have written this from the perspective of normal user applications, not specialized applications. Would you want to have a firewall programmed in Perl? Granted, you could probably do some cool things with it, but it's not exactly the most efficient language in the world. How about embedded systems in Java? Here's to hoping you can afford enough memory as some of these systems are very limited. Could you imagine doing something truly intense, like weather forcasting, in Ruby?

All in all, I think that guy was very wrong. We do need languages that are very efficient. Some things really do need as much performance as you can get out of them. If not, they won't be able to do their job as fast as needed and people are going to not want them. Granted, they may not be the most efficient for programming, but I think that's something that can be worked on as we get better libraries. For example, D has many built-in features that are very powerful and can reduce the amount of code you write.

Of course, this is coming from the person who will most likely be writing more code in D on a per-method/function basis as I'm trying to get into good habits using the language, like unit tests and design by contract, the latter of which I'm not doing so well with.

Is Drizzle the Future?

2009-05-04T18:30:00.000-07:00

I'll keep this short: are applications like Drizzle the future? I don't mean databases. I mean, instead of developing an entire app, you create a "kernel" of sorts where you can plug in various pieces of functionality to get exactly what you want. That's essentially what Drizzle has become. You can plug in a number of different pieces of functionality into it to get just the right DBMS you want. Firefox is similar to this in that it has it's extensive plugin system. There are others as well.

So, perhaps we should start thinking about creating software that serves a purpose, but can be easily extended to get just the right functionality. And, if done right, be compatible with other instances of the app or clients. This is a key as remember, Drizzle uses the MySQL protocol, so any client that can connect to MySQL can connect to Drizzle. Makes porting from one to the other easier.

Language Toolkits

2009-05-04T18:19:00.000-07:00

I think instead of learning a language or a technology, you should learn a toolkit of languages. What I mean is this: different languages are good at different things, therefore by limiting yourself to being, for example, a Java programmer, you're limiting yourself to only one way of doing things. Instead, look at the various things you may do and learn a language in each domain. Granted, some may cross domains, but the point is that different languages are better at solving problems.

For example, Perl is great for creating one-off scripts, prototyping, creating small apps where speed is less importance than making something that can be easily created, tweaked, and ran. On the other hand, D is good for things that need more speed and be friendlier on resources. Erlang and JavaScript have their own niches where they work well.

I think at a minimum, every programmer needs to learn languages that fulfill the following tasks:

Application Development (C, C++, D, Java)
Scripting/Prototyping (Perl, Python, Ruby)

For me, since I do web development and want to do more with distributed programming, I have two more types of tasks I need to know languages for. Others may have other types of tasks they need to do. The two I mentioned above I think are the most important as many applications do need the speed of a good application development language while scripting languages are absolutely fantastic at the things they do.

Food for thought.

Unit tests as design documentation?

2009-04-24T18:13:00.001-07:00

First, one thing I forgot to mention yesterday: the video card in my computer died, which is part of the reason why I wasn't blogging. The computer would have an acid-trip of sorts after a while and become unusable. I just got it replaced today with a $16 Nvidia card and it's working quite nicely. The main reason I got this card, besides the fact it was cheap, was that it uses passive cooling. God, this thing is so much quieter now...

Now, back to the main topic. After doing a bit more D coding yesterday and a little earlier today, I had an interesting thought after a couple of us talked about self-documenting code: could unit tests be used as design documentation? Think about it: if you design everything properly, your tests pass. Your tests should reflect the expected results of your methods/functions/whatever. I my case, since I'm trying to build an FSM, each test essentially documented the new state the FSM would enter based on the current state and any other inputs.

Granted, not all of your design is reflected in just unit tests, but a good portion could be. Combine this with some good documentation comments, and you've essentially provided design documentation in your code.

This is definitely something to think about because I'm thinking that the better way to document your software is in your software. If nothing else, your design is where your code is, so you have everything you need all in one place. Unfortunately, I don't know how to embed pretty diagrams in source code :-)

Update and New Direction

2009-04-23T15:58:00.000-07:00

Wow, it's been a long time since I last posted.

Anyway, as for my Erlang project, it's working! Not finished, but it does work for small test cases. I still have to work on a couple things, primarily on distributing the application and getting the data on disk, but I'm happy with the results. It handles Ands (+keyword), Ors (keyword), and Nots (-keyword). It may not be the best algorithm, but it does appear to work well. The key thing was the use of mnesia to store and inverted index to store the keywords/document IDs.

Now, since it does work, I decided to start working on learning D for a while. I've also been itching to start since pure functions are now working and I've been wanting to try out threading in it as well. Remember that Perl script I wrote a while ago? I've been wanting to rewrite it in something more efficient and D looked to be the best choice. It handles regular expressions, threads, associative arrays, etc. all build into the language. Also, I really wanted to try out some of the other cool features, such as design by contract and unit testing. I haven't gotten too far, but I'm definitely enjoying it.

The interesting thing I discovered is that creating the unit test caused me to do more design work. You see, I'm going to create a finite state machine to parse a CSV file. Well, the functions for each state will be deterministic and lend themselves to being unit tested very nicely, so I began setting up all of the different states/inputs that I'll be dealing with. During this, I discovered that my initial design was flawed, however instead of going back to the drawing board, I decided to just continue creating the tests. I think it'll work out better because I think about what I'm doing more while working with code.

You know, the more I develop using D and learn about it, the more I like it. I'm really excited about getting to try out threading. So far, I haven't created much functionality, but I'm leveraging D's debugging capabilities, unit testing, pure/nothrow functions, and documentation comments. Having all of this as part of a language instead of add-ons is pretty nice.

Now, as an added incentive for me to get this stuff done, I plan on using this for a goal at work. Hopefully I'll be able to show off what can be done in both languages because they're both pretty damn cool. That, and I'm not a fan of the trend towards becoming a Java/.NET shop. :-(

Custom Data Storage vs. Mnesia

2009-03-08T08:03:00.000-07:00

To better learn Erlang, I've been slowly working on a side project to build a search service. Since I wanted to be able to distribute the workload, my initial plan was to create a custom data structure to hold all of the data and to have each node in the cluster, with some redundancy, to hold a different subset of the data. This way, if I have to search 1,000,000 records, for instance, I can have each node look at only 100K, assuming 10 nodes, in parallel.

Well, after going through a few iterations working on the interface that will be used to access the cluster, I came to a couple realizations:

1) I don't know the best way to have a "shared" data structure in Erlang without creating some sort of artificial bottleneck.

2) Due to how Erlang works, I'd be making a copy of the data structure every time I spawn a new process.

Needless to say, I probably could have worked out number 1 in time, but number 2 is a problem. With a normal imperative language, I probably could have had a global variable that held the data in memory and, with some management code, had various processes access the same variable without making a copy of it. In Erlang, the best I could think of keep memory usage low is a singleton, which means there would be a bottleneck, one that I created. Needless to say, I don't want to make the same mistakes as people have done in the past.

Now, I didn't think about using Mnesia initially because I was concerned with having copy of the data on every node. If the dataset became large, it may be necessary to use fairly beefy machines for each node and that didn't really set well. However, I came to a realization: I don't plan on storing that much data. Yes, there may be a large number of records, but each one would be small. With disks being relatively cheap, this may not be a problem and it solves many of the other problems I was planning on dealing with, such as distributing the data. Now, I haven't read up on Mnesia too much, so I may be able to do some of the things I wanted to do anyway, but even if I can't, just using Mnesia would be a good start. If nothing else, I should be able to create something that can withstand multiple node failures and still be useful.

Regardless if how the actual implementation will work, I still plan on making it work well in a single-server environment before moving to multiple nodes. Not much of a point in building something reliable if it doesn't work right, now does it?

Perl rocks, again.

2009-02-22T18:12:00.000-08:00

As much as people have been knocking it lately, I have to say it's still a great tool to have around. I'm working on a project where we have to examine data from an external source in order to generate reasonable metrics/reports. Part of this is knowing the quality of the data. Well, I tried several tools to get this data and there were two major problems:

1) Slow. It took a while to get the metrics and typically sucked up a huge chunk of memory doing so.

2) I couldn't save the results, which meant that I had to rerun the analysis every time I wanted to see it, which meant I had a large chunk of memory being used for as long as the program was open.

Granted, this is not to say these programs didn't do a good job. They really did, at least with the analysis. Those two points are just necessary for me. If I didn't get the info in a reasonable amount of time, it just wouldn't be useful for me.

Perl to the rescue. Well, mostly. I had a script that I created before for analyzing a CSV file and making an educated guess on the data types for each column. Well, I just took that and revamped it a bit as there were some more features I wanted to add. With a combination of language constructs that make things easy to program and the wealth of modules that are available in the Perl core and CPAN, I managed to get a very nice script up and running that would analyze the CSV file, generate the info that I needed, and save the reports as HTML files.

Now, I just have to export the data as CSV, which I prefer as it makes no assumptions about types, run it through the script, and I'm good to go. Not surprisingly, this is significantly faster than running a tool that does it directly from the database. Many of the tables have over 100K rows, one had over 500K, so having to read all that data into memory and analyzing the data takes up a good bit of memory. For the analysis of one of the larger data sets, my Perl script took over 600MB of memory.

Yes, I do realize that this isn't the most efficient code for this, however I needed it sooner than later and it runs fast enough now. What used to bog down my computer for some time now may take a minute or two.

Prime Factorization Using Concurrent Processes in Erlang

2009-02-03T18:16:00.000-08:00

Well, I started doing the Project Euler problems again using Erlang and I got to one that required factorization, so I re-did problem number 3. After trying the solution advertised, I did a bit of research and I came across the Sieve of Eratosthenes. It was very simple to implement, so I used that to do the factorization. It was pretty simple and seemed to be reasonable fast.

Well, I was thinking of trying to speed up the sieve, mainly by trying to parallelize parts of it. Yep. I'm thinking more about processes as the days go on. Well, it hit me yesterday that by using the sieve to create prime numbers that I then try to use to factor a number, I could generate the prime numbers in parallel to testing to see if they're a factor. Not sure if it's any faster than the single-threaded version, especially since I'm doing this in a virtual machine, however I'm guessing that if I do get this on a multi-core machine, it should be somewhat faster.

Here's the code for anyone who's interested.


pfactor(N) ->
 SN = round(math:sqrt(N)),
 P = spawn_link(factor, pfactor1, [N, []]),
 spawn_link(factor, pseive, [lists:seq(2, SN), P]).

pfactor1(N, Factors) ->
 io:format("In pfactor1.~n"),
 receive
  {factor, Factor} ->
   io:format("Received a factor: ~p~n", [Factor]),
   if
    (N rem Factor) == 0 ->
     pfactor1(N, Factors ++ [Factor]);
    true ->
     pfactor1(N, Factors)
   end;
  {done, []} ->
   io:format("Done.~n"),
   io:format("Factors: ~p~n", [Factors])
 after 10000 ->
  io:format("Timed out...~n")
 end.
 
pseive([P | R], Pid) ->
 io:format("Sending the new factor: ~p~n", [P]),
 Pid ! {factor, P},
 pseive([X || X <- R, (X rem P) > 0], Pid);
pseive([], Pid) ->
 io:format("All done...~n"),
 Pid ! {done, []}.

Thinking in Erlang

2009-01-18T16:07:00.000-08:00

First, I'd like to say that my previous post was somewhat incorrect: there are MapReduce solutions written in Erlang. The great oracle, Google, has told me so. However, the ones I found are not quite what I envisioned. They all appear to be single-machine solutions, not distributed solutions. I'd like to see a multi-machine MapReduce solution written in Erlang. It looks like it could do it real well.

Anyway, the real meat of the post I is how I think when I'm thinking about writing code using Erlang. When dealing with databases, I think about how to structure the data so I get the results I want. Tables, columns, relationships, etc. With programs, I think about functions and data types. In those cases where I'm working in an object-oriented environment, I think in terms of objects. For web sites/applications, I think in terms of pages or interfaces, when dealing with AJAX requests.

Erlang is different. Instead of primarily thinking in terms of functions, data, etc., I'm thinking of in terms of processes and machines. I guess it's the fact that you can easily create and communicate with processes that it just becomes natural to think in terms of them. I'm trying to think of how to do a MapReduce style application and my first thoughts deal with how I'm going to try to distribute it across multiple machines and have them communicate with each other in a sensible fashion.

WTF?

Yeah, it's new to me. It's...interesting. I've never thought about applications like this before and it's an interesting experience. Granted, I may be biting off more than I can chew, but I think it's a reasonable starting point. Since this is an application that I want to work across multiple machines, it seems that the first thing that needs to be worked out is how the different machines communicate with each other. I know I can communicate with processes across multiple machines, but how do I do it in an organized fashion.

Needless to say, I'm intrigued.

MapReduce with Erlang?

2009-01-12T18:23:00.001-08:00

A funny thought hit me yesterday before I forgot it and re-thought of it today: is Erlang the ideal platform for MapReduce applications? Let's see...

Highly concurrent, even across multiple machines.
Is functional, therefore it does have map and "reduce-like" functions.
Designed for high-reliability, including the ability to upgrade without shutting down.

Interesting, no? It's definitely something to think about and I think if I do get a chance, I'll have to see what I can do along those lines.

Perhaps I should suggest that the Computer Language Shootout should have a MapReduce benchmark...

What can a broken toilet teach you?

2009-01-11T18:48:00.000-08:00

A couple days ago, a toilet in my house cracked and leaked water on the floor of our bathroom. My wife and I got a replacement yesterday and I managed to get it installed pretty quickly, so it was not too much of a deal. However it got me thinking a bit. This makes a good computer analogy.

You see, we got lucky. We have been working on our secondary bathroom off and on, mainly off right now, since August, and we got the old toilet re-installed and working. This worked out well as I didn't have to resort to writing my name in the snow. How does this apply to the computer world? Having one of something is nice. Having a backup is a good idea when it's a critical system. :-)

Creating More Reliable Software With Good Concurrency

2008-12-28T16:27:00.000-08:00

First off, hope everyone had a good holiday season.

Second, something has been on my mind of late: creating better quality software that has good concurrency. You see, I'm working on a system right now where high availability is a major concern, at least for me. Also, with multi-core systems becoming the norm, it would be good to be able to take advantage of these systems without running into more problems.

I have been contemplating the for a while and even though I feel I am good enough to do it in just about any language, I'm a bit nervous about doing it correctly, at least when it comes to concurrency. It appears that there are a variety of solutions and not all of them work as well as we would like them to. Also, to me, the language should make it easy to do this without the programmer having to ensure they don't make a mistake. This isn't quite like C where accidentally using = instead of == causes unexpected behavior. In that case, C is allowing you to be more expressive if you choose. Do it incorrectly and yes, you'll shoot yourself in the foot. Use it correctly, and your code can be more compact and elegant. There's a benefit to doing things in the not-so-clear way. With concurrency, I don't see a benefit to doing any way but the correct way.

Now, concurrency from what I can tell comes in two different forms. First, is threading/forking where you run multiple "processes" on the same machine. This is what I was talking about above and is considered a difficult process. This is mainly because of the sharing of data between "processes." The second type is running a job over multiple machines. The same issues hold true, but appear to be more easily managed. I did this on a project, but I relied on a central database to manage the jobs. By leveraging transactions, I ensured that no two machines worked on the same set of data. This works great, as long as your central server is up and running. Clustering would remedy this, but this doesn't mean it's a good general case solution for all problems. For what I was doing, this worked perfectly because I was expecting each server to perform some long running processes, therefore the overhead of connecting to a central database had a minimal impact on performance.

Making more reliable software is much easier, though we as an industry still fail at it. We have many tools, such as unit testing libraries, static analysis tools, and debuggers as well as techniques for improving software quality. My concerns are more along the lines of high-availability. This doesn't just mean software bugs, it's also about ensuring that if something beyond the programmers control goes wrong, the system can try to remain available.

Initially, I was looking into D to help with this. It has a number of feature that help with this, however I'm not 100% sure the threading model is ideal with respect to keeping me out of trouble. I also looked into Haskell which essentially forced you into doing good things by virtue of it being functional and it's use of software transactional memory is impressive, but it wasn't quite what I was looking for, though I didn't realize it for some time.

I finally found what I think is the right answer: Erlang. You see, the more I read about it and tried to understand it, the more I think it's what I'm looking for. Now, I still have to actually develop some code in it and understand it some more, but the design seems to be just what I want. Each "process" is completely independent of each other and communicates to other "processes" through message passing. Therefore, there are no locking issues, which seems to make things easier. Also, these processes can communicate over multiple machines without any special coding, or at least that's how it seems.

Erlang is also supposed to have very good error handling as it was designed for systems that require high availability. I haven't looked into it yet, but I'm hoping it's pretty easy as well. A good feature that I definitely will be exploring is it's ability to be upgraded in place. This is very nice for when you have to perform an upgrade, but you can't take the system down for very long if at all. Needless to say, these interest me, especially in when I look at what I'm doing at work.

My plan right now is to first try out the communication between processes on a single machine, then across a couple. Not really sure how I'll pull that off other than I may just have to run a couple VMs sometime. Then I'll look into the error handling and upgrades. I'm seriously considering working on a project using Yaws sometime, but that's another post. Web applications are probably where this is going in terms of projects, but I don't know if I'll get to do it at work.

My main concern is database connectivity and integrating with C libraries. The people behind Erlang seems to be very honest about it not being the best tool for every job, which I appreciate, hence why they designed it to access C libraries. In fact, that may be how I end up communicating with a database. Not really sure as there are several libraries for Erlang that do it already. I just don't want these to be the cause of a performance issue.

Anyway, wish me luck in my endeavor as I'm doing this in part to improve software quality at work and in part to make me a better programmer. Oh, and Happy New Year!

Portability via Stored Procedures

2008-12-03T04:01:00.000-08:00

This was a thought that just hit me yesterday. If you encapsulate all of your queries into stored procedures, I think if you build your app correctly, it can increase the portability of your application. See, if you use something like Perl's DBI, you can treat every RDBMS identically from an API perspective. So, if all you do is call a stored proc, all you should have to do is change the driver/connection string and you're done. Your stored procs can hold all of the queries and still allow you to use vendor-specific features to be as optimized as possible. Granted, you still have to maintain a set of stored procedure scripts per vendor, however chances are you'll have to have to maintain SQL scripts anyway for table creation, tablespaces, etc.

I bring this up because in every discussion I've seen related to databases, stored procedures, and portability, I've never seen this suggested.

Current MySQL Release Comments

2008-12-03T03:51:00.000-08:00

Well, MySQL 5.1 has been released and it's apparently not up to par with previous releases. While we all know that no software is completely bug free, unless of course you're talking about NASA (WOW!), apparently some major bugs did slip through. Granted, these are edge cases if my reading is correct, however it is cause for concern.

So, does this mean that the anti-MySQL should rejoice and say "Hah! Ours is better!" No. First, there are still things you can do with MySQL that should still work just fine and can't be done with other RDBMSs. Second, your systems have their problems as well. While vendors like Oracle, and I think even the PostgreSQL people to a degree, would love to say that theirs is the end-all, be-all of databases, they aren't.

Now, how does this impact my view of things? Well, it was kind-of a shot in the arm to maybe look into other solutions a bit more. Yes, I most likely will still be using MySQL for some projects, but oddly enough, I'm now have an interest in PostgreSQL, mainly for some of the more programmatic things it can do. It has its warts, but I think it brings a lot of nice things to the table. Also, I'm now more interested in Drizzle.

Maybe I should look into Firebird as well. Anyone know what the benefits are of it compared to PostgreSQL?

Functional Programming for the Web

2008-11-12T19:29:00.000-08:00

Ever since I saw a link on reddit for a functional language that was designed for web programming, I became interested in the idea. Not really sure why I waited until then as I was playing with Haskell for a while. It looked like someone was going the pure functional programming route similar to Haskell and Erlang. I even looked into Erlang and Yaws, but wasn't impressed with what I saw. None of it looked like how I'd want to program the web.

Here are my current thoughts:

Have it be a hybrid language, perhaps similar to D. One thing I noticed with both functional and procedural languages is that some problems are more easily solved in one vs. the other, so why not take the strengths of both and leverage them?
Treat IO, such as the results from a database query, as streams that can be acted upon using map/reduce/foldr/whatever functions. To me, this would make things very clean.
Be lazy. Web programming can be described as processing some data and sending the user a response in a short timeframe, so why not only do what needs to be done and ignore what doesn't need to be done?

Now, I haven't figured it all out, and may never do so, however I think this will work nicely. The thing to remember is that web programming is ideally used for applications with very simple data processing needs. Nobody wants to wait for a web page to load and web servers really aren't meant to do real data processing. It shouldn't be impossible to do so, but the language should be focused on making it very easy to do the most common things, but possible to do the less common.

Everyone else is doing it, so why not me?

2008-11-12T19:27:00.000-08:00

Instructions:

Grab the nearest book.
Open it to page 56.
Find the fifth sentence.
Post the text of the sentence in your journal along with these instructions.
Don't dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

"The strcat() function concatenates a string to the end of a buffer." -- Secure Coding in C and C++ by Robert C. Seacord.

I am such a nerd.

Looking for work

2008-10-24T18:25:00.000-07:00

Blogging is slow as I'm currently looking for work. I'm out in western PA, so if anyone out there is interested, let me know.

Related to this and needs around the house, I haven't done much in the way of programming at home. However, I did start to work on a little C coding. Figured I'd best make myself a bit more marketable :-) So, I decided to start by writing a string library that would not be affected by buffer overflows. So far, I just got a strcat replacement working. It appears to work great, however since I use malloc to allocate a new buffer that contains the two strings concatenated, the new string has to be released using free. It's a bit more work, but it may be worth it. Of course, it would be nice to not have to worry about it. Just not sure how...

However, I think if I get the library working the way I want, I can later port it to C++ as a class. That may take care of the malloc issue since I can rely on a destructor being called to release any allocated memory.

When I get the code back to my home computer, I'll have to post the functions.

Testing Tips

2008-10-01T18:12:00.001-07:00

I've been doing a good bit of testing of late for the project I'm working on thats now needs to be completed by COB Friday. I figured I'd post a couple things that I learned that may be useful.

1) The Gimp is your friend. Well, you don't need the Gimp per se, but an image manipulation program is a good thing to have. Trying to describe a bug is one thing. Make a screen capture, annotating it, and sending it to someone is a whole other thing. You really don't need to learn how to do much to be effective. Draw a box/circle, put text on the image, maybe draw a line or two, crop, and manage layers. That's it. I think that's less than 1% of what any decent image editor can do, so there's no reason to get Photoshop or something expensive.

2) Ad-hoc as you go. First, it can be tough to get a good test plan. Due to our schedule, or lack thereof, our test plan isn't that great because a lot of things don't get tested. So, test them as you go along. If there's an option that can be changed and the test plan doesn't say to do anything with it, change it. A user may do it so you should know if the application can handle it.

3) Stress test the damn thing. You don't know what user's expectations are, so throw more data at it than you think it will handle and see what happens.

4) Get good tools. For web testing, there are a variety of extensions for Firefox that are good for helping you generate data, resize your browser, see what errors are being thrown, etc. True, you don't need to use them, but if they make your life easier, why not?

5) Get good error information. Don't be vague. Don't just say "feature A failed." Get good info! Look at logs. Capture error messages. Sometimes when the developer gets around to trying to fix the bug, the log may be gone or you forget how to reproduce the error. Yes, it takes a little extra effort, but it means that the developer can get the bug fixed faster.

Enjoy.

Exception Handling Thoughts

2008-09-15T18:01:00.000-07:00

I have discovered over the past couple months that people seem to like to wrap exceptions around larger "units" of code, such as the catching the exceptions in the controller of an MVC framework where the controller can call many functions underneath it. I have one thing to say about this: I don't like it.

There, I said it. To me, it seems very counter-intuitive to do this. If a method throws an exception, shouldn't you catch it right away? That's what I always thought. Now, people may argue about why you should do that and that you can't handle errors at such a low level, however I do believe I have one important reason to catch them as early as possible: state.

You see, how do you know what was passed into a method at a low level when you're 6 levels above it? Easy, you can't. Granted, I do see benefits in capturing exceptions at a higher level. Mainly, I see that it's a consistent location to display messages to the user. However, at the lower level you can do the following:

Determine what the error really is with better accuracy.
Log the state of the system at the time of the error.
Handle the error appropriately, where "handling" the error may be retrying the code or simply rethrowing the error.
Unit tests are performed at a lower level, so you can't rely on the high level exception handler to do anything useful with an exception. Capturing exceptions sooner than later means you don't have to wade through stack traces to figure out what happened.

Now, another thing you can do is capture exceptions all the way up the to the highest level. This way you can capture very detailed information about the state of the system and know exactly how what path you took through your program.

Why am I so upset with the current exception handling situation? Easy: I hate Java stack traces. They are god awful to read and at times tell you nothing useful. If someone where to put exception handling in their code at a lower level, I can get a better idea of what went wrong provided that the user did their job properly. I have enough problems with Java since I don't code it in very often so don't make my life harder.

Status Class

2008-09-03T19:10:00.000-07:00

Well, I made some progress on a project I'm working on. The I/O portion still needs work, but I did create something that is somewhat interesting: a status class. What I wanted to do was have the ability to record status information about any errors/failures/successes/etc. that occur when a member function is executed. I got the idea while reading some recent articles on exception and why they're bad, how they're misused, etc.

It's very simple. First, you have a set of states that represent the state of the recently executed function. Success, EOS (End of Stream), and Error are three of them. You can also set a message, which you need for an error. Now, there is a consistent way to check the status of a function and know what happened without relying on exceptions or special return codes.

Originally, this was supposed to be a helper class where it would encapsulate the result of the function as well as the status information. Now, I have it set up to be a base class that other classes can inherit from. Much cleaner and I think it'll work out quite nicely.

Now, back to getting my I/O module up to snuff.

Back and Unit Tests

2008-08-13T19:06:00.001-07:00

First, back from vacation. Was relaxing until I got back to work and I was greeted with "PANIC!!! WE NEED THIS NOW!!!" Yeah.

Second, I started a new project mainly as an attempt to learn D. So far, I haven't gotten very far, but I do have a makefile setup so that I can build the project with or without unit tests enabled. D has an interesting feature with the compiler where you can build your executable with unit tests enabled and they are run when you execute the program. So far, it seems to work very well. Would have continued but I ran into a problem: data loading.

See, I want this program to work on large data sets that won't necessarily fit in memory, so if I write a function that is supposed to work over a large set of data, I can't just pass in an array and go forth. Ah well, it's probably best that I start there anyway. I just wanted to get the unit testing working so that I fully understood how it worked.

Improved Security by Switching Languages?

2008-07-28T15:41:00.000-07:00

Had a thought recently and I figured I'd post it. How many of C/C++'s issues in terms of security are resolved by switching languages? For example, D has a string type that you can put variable-length strings into. Will using that instead of a character array will that solve if any? It's an interesting thought because if the language provides better security through it's design, then perhaps we can focus more on the actual problem we're trying to solve instead of having to remember when we can/cannot use strcpy without cause our application to be a huge security risk.

Using a functional programming language will probably solve many of these problems, but I'm wondering if D may be a good-enough replacement since it still allows you to do C/C++ like things, just in a safer fashion.

I have the O'Reilly Secure Programming Cookbook for C and C++, so I may re-read it and see what I come up with.

D 2.0 Rocks!

2008-07-23T18:11:00.001-07:00

Probably not as much as you think, but this just tickles me pink. I rewrote a simple factorial program I had written to use Design by Contract and Pure functions. So, what does this mean?

First, Design by Contract allows me to check the input and output values to ensure that they're correct. In the case of the factorial function below, I check to see if the input and output values are both greater than zero. It's kind-of pointless for the output, but it didn't hurt to put the check in.

Second, a Pure function is one that behaves in a functional manner. In other words, no side effects. You pass data in and you get a return value. You can't use global or static variables. I think this was originally done for threading, but I like it because you can make functions safer all around. Think about it, no worrying about the function changing a global variable somewhere accidentally. These are restricted to a degree in that pure functions can only call pure functions, but you can use "non-pure" code inside of a pure function, such as a for loop. This way you get the niceties, if not the best, of both worlds.

Below is the code. Enjoy.


import std.stdio; // Module for console IO.

pure long calcFac (invariant int value)
in
{
    assert(value > 0, "Input value is not greater than 0.");
}
out (result)
{
    assert(result > 0, "Return value is not greater than 0.");
}
body
{
    long retval = 1;
    int i;

    for (i = 1; i <= value; i++)
    {
        retval *= i;
    }
    return retval;
}

int main(char[][] args)
{
    try
    {
        writefln("Result: %d",calcFac(4));
        writefln("Result: %d",calcFac(0));
    }
    catch (Exception e)
    {
        writefln("Caught: %s\n", e.msg);
    }

    return 0;
}

Yes, it is a bit wordy for a simple factorial, but this one the first idea I had to try this out with and I think it's a good one.

Project Euler Going Well

2008-07-10T19:07:00.000-07:00

Just a quick post today. Lately, I've been using Project Euler for programming exercises and it's been fun. It's interesting to compare and contrast my solutions written in D and Haskell. Each one is nice in it's own way and one is usually more elegant than the other. Sometimes it's D. Other times it's Haskell. Either way, it's been fun.

Right now I'm on problem 9 out of 200 or so. Wish me luck!

Geo-conversion finished, for now.

2008-06-29T11:53:00.000-07:00

Well, I got my little geo-conversion program finished. Nothing too fancy, but it works. I think I'd like to see if I can use a arbitrary-precision library to do the calculations instead. Right now, everything is double, but I'm not sure how much it matters. Hell, I don't even know how many digits past the decimal point anybody really cares about.

Anyway, here's the code:


import Text.ParserCombinators.Parsec
import Data.List

-- This data type stores the three pieces of info we need from each line.
data ConvLine = ConvLine {conversion::String, value::String} deriving Show

-- This function parses each line of input.
parseLines = do
        lines <- many inputLines
        eof
        return lines

-- Separate each line of the input into two parts: the conversion to be
-- performed and the value to convert.
inputLines = do
        conversion <- many1 (letter <|> char ':')
        spaces
        value <- anyChar `manyTill` newline
        return (ConvLine conversion value)

-- This does the conversion.  Using the whole conversion key as the different
-- cases.
geoConvert :: [ConvLine] -> [String]
geoConvert l = map doConvert l
        where doConvert s = case (conversion s) of
                "DMS:DD" -> dmsToDD (value s)
                "DD:DMS" -> ddToDMS (value s)
                _ -> error $ "Undefined conversion: " ++ show(s)

-- Convert DMS to Decimal Degrees.
dmsToDD :: String -> String
dmsToDD s = case (parse parseDMSToDD "" s) of
                Left err -> error $ "Input:\n" ++ show s ++ 
                                    "\nError:\n" ++ show err
                Right result -> show result

-- Convert Decimal Degrees to DMS.
ddToDMS :: String -> String
ddToDMS s = case (parse parseDDToDMS "" s) of
                Left err -> error $ "Input:\n" ++ show s ++ 
                                    "\nError:\n" ++ show err
                Right result -> result

-- This one is simple.  First, break apart the different pieces of the value
-- into their numeric components.  Next, calculate and return the Decimal
-- Degrees value.
-- Key point here: read takes a string and converts it to a number if
-- possible.
parseDMSToDD :: Parser Double
parseDMSToDD = do
    degrees <- many1 digit
    char 'D'
    minutes <- many1 digit
    char 'M'
    seconds <- many1 digit
    char 'S'
    return ((read degrees) + ((read minutes) / 60) + ((read seconds) / 3600))

-- This parses out the number from the value and passes it to the calcDMS
-- function.
parseDDToDMS :: Parser String
parseDDToDMS = do
    decimalDeg <- many1 (digit <|> char '.')
    return (calcDMS (read decimalDeg))

-- This function creates the DMS string.
calcDMS :: Double -> String
calcDMS dd = (show (getDegrees dd)) ++ "D" ++
             (show (getMinutes dd)) ++ "M" ++
             (show (getSeconds dd)) ++ "S"

-- This function gets the degrees part.
getDegrees :: Double -> Integer
getDegrees n = (truncate n)

-- This function gets the minutes part.
-- Key point here: need to use fromInteger to convert from an Integer to a
-- Double.
getMinutes :: Double -> Integer
getMinutes n = (truncate ((n - (fromInteger $ getDegrees n)) * 60))

-- This gets the full value for the minutes.
getFullMinutes :: Double -> Double
getFullMinutes n = ((n - (fromInteger $ getDegrees n)) * 60)

-- This function gets the seconds part.
getSeconds :: Double -> Double
getSeconds n = (((getFullMinutes n) - (fromInteger $ getMinutes n)) * 60)

main = do
        -- Read from stdin.
        input <- getContents

        -- Get the result of the parsing.
        -- Left == error
        -- Right == success
        let convLines = case (parse parseLines "stdin" input) of 
                Left err -> error $ "Input:\n" ++ show input ++ 
                                    "\nError:\n" ++ show err
                Right result -> result

        -- Do the conversion.
        let outLines = geoConvert convLines
        print outLines

My only complaint is how I had to do the conversion from Decimal Degrees to DMS. The various parts of the process are reusable, however I really wish I could have gotten it to work in only one function. When I tried it, the "return ..." statement at the end kept throwing a compilation error. Oh well, I'm probably just missing something. I'm sure I'll figure it out later.

UPDATE:

O.K., I really shouldn't post when I'm not thinking clearly as I still have at least one thing to do: convert to/from radians. I was also planning to convert between lats/longs and UTM/MGRS grid coordinate systems, however those are much more complicated to do, so I will most likely be putting them off for a while. I may have to deal with them at work, but until then I don't want to spend a lot of time on them since the real purpose of this exercise is to learn Haskell and not how to do the conversions.

Another note, as to how this program works, it reads from STDIN and outputs to STDOUT. I'm expecting the data to be stored in a text file or it can be streamed it. The format is very simple:

operation value

Where operation is "DMS:DD" or "DD:DMS" and the value is the value you want to convert. Currently, only the 1D2M3S format for Degress Minutes Seconds is supported for simplicity.

As I'm still under the weather right now but unable to sleep, I bid you all farewell and good night.

Converting Geosptial Coordinates

2008-06-24T18:02:00.000-07:00

Well, I've been focusing on Haskell of late mainly because there's more for me to learn. As I believe I stated before, D is similar to what I've learned before, so it's easier for me to pick up. On the other hand, I've been spending more time getting used to the way Haskell works.

One thing that I'm itching to do is create a useful little program right now. I just read several sections in Haskell book/tutorial on wikibooks and at least one other tutorial, so I think I know almost as much as I need to know. The problem was, what type of program to create.

At work, we're dealing with some geospatial data and the one thing I learned yesterday was how coordinates are represented in some grid-based coordinate systems. During the research we were doing, the guy I'm working with found a link with a number of formulas for converting between the different coordinate systems. Score! I can make a small app that reads from stdin, converts a list of values from one coordinate system to another, and outputs them to stdout. This way, data can be piped through easily.

I know this isn't the most useful app, but it seems like just the kind of thing I would want to do in a language like Haskell since we want it to work right every single time without fail. Now I just have to define the input/output formats and see how much I like Text.printf :-)

Area/Volume Programs

2008-06-16T19:03:00.001-07:00

Well, over the weekend and today, I created two programs that calculate areas. I'll put them below, but for now, I just want to point out some things I observed.

First, the Haskell tutorials are still giving me trouble. I'm doing a little better, but I had a bit of trouble trying to concatenate a number to a string. I thought it could be done with the ++ operator, but no, it would fail. It turns out you need to call a function called show to turn a number into a string or something like that. I never saw it in any of the basic tutorials.

Second, the compile time of GHC is very noticeable now. Here are some numbers:

GHC:
real 0m1.370s
user 0m0.472s
sys 0m0.104s

GDC:
real 0m0.243s
user 0m0.104s
sys 0m0.020s

Big difference for such a small program. Not really sure why this is happening, nor do I think I care too much right now. Of course, as programs get bigger, so do compile times. I wonder if this discourages some users from using a language like Haskell.

On the bright side, I'm liking Haskell's math capabilities more and more. You see, I wrote the Haskell version of this program using a tutorial. Well, I went beyond the tutorial, but that's besides the point. Tonight, I wrote the D version and I compared the outputs. It's very interesting to see the difference in precision that you get from the languages. Haskell is more precise and the runtime doesn't seem any different, thought it appears there is a difference:

GHC (ghc -o a area.hs):
real 0m0.004s
user 0m0.000s
sys 0m0.000s

GDC (gdc area.d):
real 0m0.015s
user 0m0.000s
sys 0m0.004s

The numbers are really to small to really tell a difference and I didn't do any optimizations, so who knows if this is accurate. I put the commands I'm using to do the compilation in parenthesis in case someone is curious.

All in all, I think I'm going to stick with the current pattern of doing something in Haskell first, then in D. I can understand and write code in D much faster than Haskell, so I figure I'd learn the "harder" language first, then write code in the "easier."

Anyway, here are the Haskell and D versions of my program:

Haskell:

{- Calculate the area of a circle. -}
circarea r = pi * r ^ 2;

{- Calculate the area of a triangle. -}
triarea b h = (b * h) / 2;

{- Calculate the area of a rectangle. -}
rectarea l w = l * w;

{- Calculate the area of a square. -}
sqarea s = rectarea s s;

{- Calculate the volume of a box. -}
boxvolume l w h = h * (rectarea l w);

{- Calculate the volume of a cylinder. -}
cylvolume r h = (circarea r) * h;

main = do
      {
          {- Apparently, the "show" function is required to convert a number
           - to a string? -}
          putStrLn ("Circle Area: " ++ (show (circarea 2)));
          {- print and putStrLn behave differently.  Observe the output. -}
          print ("Triangle Area: " ++ (show (triarea 2 2)));
          putStrLn ("Rectangle Area: " ++ (show (rectarea 2 2)));
          putStrLn ("Square Area: " ++ (show (sqarea 2)));
          putStrLn ("Box volume: " ++ (show (boxvolume 2 2 2)));
          putStrLn ("Cylinder volume: " ++ (show (cylvolume 2 2)));
      }

import std.stdio; // Module for console IO.
import std.math;

real circarea(real r)
{
  return PI * pow(r,2);
}

real triarea(real b, real h)
{
  return (b * h) / 2;
}

real rectarea(real l, real w)
{
  return l * w;
}

real sqarea(real s)
{
  return rectarea(s, s);
}

real boxvolume(real l, real w, real h)
{
  return rectarea(l, w) * h;
}

real cylvolume(real r, real h)
{
  return circarea(r) * h;
}

int main(char[][] args)
{
  writefln("Circle Area: %f", circarea(2));
  writefln("Triangle Area: %f", triarea(2, 2));
  writefln("Rectangle Area: %f", rectarea(2, 2));
  writefln("Square Area: %f", sqarea(2));
  writefln("Box Area: %f", boxvolume(2, 2, 2));
  writefln("Cylinder Area: %f", cylvolume(2, 2));
  return 0;
}

First set of lessons/code

2008-06-14T17:42:00.000-07:00

Well, I just wrote a few very simple programs and it's been interesting. I did a "Hello World" in both. D was very easy, however it took a while for me to figure out Haskell as most tutorials are done using an interpreter, which apparently is very different from using a compiler, which is what I was trying to do. I'm personally not liking this aspect of the language because I don't know of any other language that does that. Perhaps it's just the quality of the tutorials, but I think that code written in an interpreter should be identical to what you have to compile using a compiler.

Anyway, I have a factorial program written for each. Well, I have two for Haskell as there are two very different ways according to the one tutorial I was looking at. I'll just show you the cleaner one.

First, D:

import std.stdio; // Module for console IO.

int main(char[][] args)
{
  int value = 4;
  int factorial = value;
  while (--value)
  {
      factorial *= value;
  }
  writefln("Result: %d",factorial);
  return 0;
}

And now Haskell:


{- Factorial when input == 0 -}
fac 0 = 1

{- Factorial in all other cases -}
fac n = n * fac(n-1)

main = print (fac 42)

Both are very easy to understand and clean. The biggest difference between the two isn't the syntax, but the result. In D, the factorial of 42 was too big for the integer variable to handle, so it kept returning 0. However, Haskell easily printed out what I'm guessing is the correct result. This threw me off because I was expecting to see a similar result from the D version instead of 0.

While I haven't fallen in love with Haskell yet, mainly because it's very different from what I'm used to, I do like the fact that it does that be default instead of requiring an external library that may or may not integrate cleanly with the language. I'll have to read up on it some more, but I think that it'll be nice for applications that need arbitrary-precision math capabilities.

Going back to the syntax, the Haskell one is much easier to read in this case, but this is a basic math problem, so I would expect this. I do know that it'll take me some time to be comfortable with it, but I'm definitely willing to push forward and learn as much as I can stand, especially if I can use it to make reliable apps that don't need C-like speed, but do need something better than Perl.

Learning new langauges

2008-06-11T18:26:00.000-07:00

Well, I've finally decided to learn a new language. Two actually. D and Haskell. Here's the way I figure it. I already know a good scripting language: Perl. While Lua interests me because of it's embeddable nature, I couldn't see anything I would use it for personally. Perl has many more useful libraries. Also, since I know Perl, I don't see the need to learn Python, Ruby, or any other scripting language because I don't see any reason to do so. They don't provide anything that I can't already to in Perl and if I did have to work with any of them, I'm pretty sure I can pick up a reference, look at a few tutorials, and work through any code I need to.

Now, why D? Easy: I wanted to learn a C-like language and D looked the best to me because it doesn't make me have to deal with character strings. What I mean is, I don't have to allocate/deallocate/check buffer lengths, etc. Having a native string type is very nice to me. Also, it has a variety of nice features that look pretty good and it looks to be well designed. Not perfect, but good. Also, since it can work with C libraries, I can resuse a lot of existing code. The only thing I lose compared to C is a little performance, but I think it's worth it at least for the added safety of the language.

Now, why Haskell? Simple: I feel the need to learn a functional programming language. After looking at it a bit more, it seems to have a nich between D and Perl in terms of performance and programmability. If I need speed, I can use D. If I need to come up with a quick script, I can use Perl. If I need to write a good app, I can turn to Haskell for something that's faster than Perl, but potentially, I'm not sure yet, safer than D and hopefully easier to code in. Also, I think that learning how to program in a functional language would be good for me.

Anyway, now the big problem is time :-)

ParrotOS

2008-05-16T19:55:00.000-07:00

Wow, a whole month with no updates. Well, I've been very busy with stuff around the house and at work, so I guess that's to be expected.

To the point of the post, is it time for ParrotOS? ParrotOS is my nickname for an operating system that's a virtual machine. All machine-specific instructions/configurations/etc. are hidden from the programs running on the machine by the kernel. Also, the reason I call it ParrotOS is because I can see the Parrot Virtual Machine being a good choice for this.

Now, what's the point of this.

Since all software would be compiled for one architecture, the VM, you can go from machine to machine and never have to recompile a binary.
The kernel could be made aware of hardware optimizations and take advantage of them if possible. For example, if we have a math co-processor that's much more efficient for scientific computing applications, the kernel could take advantage of that for any instructions that perform such operations. This can result in a nice speed boost for an application without knowing what the hardware is.
In theory, if the "machine code" is well defined, you could write an app in any language, compile it, distribute it, and if someone wanted to modify it in their own language, they could decompile the binary into that language. For example, I could take a program written in C and decompile it into Python to modify it. Granted, this may be too theoretical, but still cool.
This could potentially be a faster OS and previous generations since there are many optimizations you could do in the background, such as good garbage collection and optimizing apps on the fly.
Potentially better security by only allowing safe operations to be performed in the VM.
Eliminate language performance differences. Instead of having to focus on the "performance" of a language, you can focus on other, potentially more important aspects, such as functional vs. procedural.

Now, this may force people to move away from certain languages as they may not work well with the kernel. For example, if our VM of choice uses a very safe string data type, then programming in C may not work out the best since there a string is an array of characters.

Anyway, I'm thinking more and more that this is the time to start working on such a beast. I think more consistency across operating systems is going to make things a lot easier on everyone and if we can have vendors like IBM and Sun talking the same language, we don't have to worry about porting issues. The software treats them both as the same machine!

The big problems are vendor support for drivers and ensuring reasonable performance for heavier apps, such as a GUI environment and office suites.

Loving Perl

2008-03-27T14:53:00.000-07:00

Oh yeah. My work day ended on a high note. You see, one of my coworkers needs to be able to take large amounts of delimited data and convert it to XML, or something similar. So, I got to spend some time earlier this week and late last week working on a Perl program that will take an input file and export it using a template. It works well and he even likes it.

Today, he asked if I could look at some more complicated data sets. By virtue of good design and the ease of making coding changes, I was able to get two of these data sets into a new format while fixing bugs and creating an extra helper script in less than two hours.

One of the other cool things is that I created a script this morning that performs some cleanup on XML files, namely removing empty tags and blank lines. I built it as a package that could be run from the command line, which is a nice bit of flexibility. I got to actually import it into the program described above and it worked like a charm. Not perfect, but pretty close.

Anyway, hopefully tomorrow I can make some improvements to these helpers scripts/packages and get the last data set taken care of.

Perl rocks!

Elive no more...

2008-03-23T18:31:00.000-07:00

Well, at least not for my desktop. I got sick of not having newer versions of some packages, so I decided to try out something Ubuntu-based. I first tried OpenGEU, however it kept trying to run something gtk+ related as SUID and failing. This was disappointing because it looked great if not somewhat buggy off the livecd. Since I prefer a lightweight desktop environment, I tried Xubuntu and it's working well. Not quite perfect, but I can live with it. It runs fast and doesn't get in the way.

I did try to install Enlightenment, however it was DR 16 and after using DR 17, it's not very nice. I tried to launch Gnome, but I ran into the same gtk+ problem. No idea why.

Anyway, so far it's working out very nicely. The XFCE desktop works very well. Hell, even the PasswordSafe try icon shows up in the icon tray.

Cool Template Package

2008-03-23T18:29:00.000-07:00

I recently found a new template package called Template::Recall. Dear lord, it is nice. You have complete separation of presentation, like HTML, and your Perl code. I used it in a piece of software at work to convert from a tab delimited format to XML and it works great. Very easy to set up and get working.

For more, search for it on CPAN.

Pcal in your browser

2008-03-13T16:29:00.001-07:00

Recently I installed and set up a calendar using pcal. While I like the postscript output, I came up with a good idea: why not make my homepage the HTML version of the calendar? So, what I did was I set up my calendar file to include all important holidays and some birthdays. Then, I wrote a quick script:

pcal -H -o ~/cal.html

Now, I made this my default homepage. I also wanted to have it be my default page in new tabs, which is needed since I tend to leave tabs open, so installed the New Tab Homepage extension for Firefox.

The last step was to set up a cron job so that a new calendar would be created at the start of every month. This hasn't been tested yet, but it should work.

So, now whenever I open a new tab, I see what events are occurring this month. Simple, but effective. Gotta love good tools. And Linux :-)

Yay!

2008-03-02T18:52:00.000-08:00

Well, I finally did it. I installed Linux on my desktop. After years of having it on for when I played games, I can finally move on. All I have to say is DAMN! It feels so bloody fast, it's insane.

Btw: I use Elive as my distro of choice. A nice blend of lightweight and powerful. That, and Enlightenment is the best windows manager you can get :-)

Revolutionary!...er, no

2008-02-18T13:54:00.001-08:00

I was thinking about a presentation I was working on about my use of LaTeX for a specification document at work and I realized something: CSS really isn't all that cool. See, I was thinking about the fact that with LaTeX I can put document styles in a separate file that I can reuse for different documents. CSS does the same thing and when I first came across it, it was really cool. Now I'm wondering how many other languages/document formats support this.

This also reminds me of language debates. People are saying that I need to learn Python or Ruby or something else. My response to this is "what do they do that Perl/PHP/other, which I know already, can't do?" There are many languages that all solve the same problems. They just do it in different ways. Now, some languages have interesting features or specializations, but one or two features is not enough for me to just learn a new language.

I think I'm becoming a real cynic because I can apply this thought process to a number of different web sites/applications as well. Is Google's search really revolutionary? No, not really. I remember AltaVista having a similar interface. Granted, Google provides better results than most, but I'm thinking it's more evolutionary than revolutionary. Google's MapReduce? Again, no, not revolutionary. It's based on language features that existed for years in languages like Lisp. They're just applied in a different fashion. A site like Digg isn't really all that special in reality because when you really look at it, it's just a list of links to other sites that are rated by users. Whoopee. People have had lists of links to other sites since the inception of the World Wide Web. All a site like Digg did is provide a rating system that works reasonably well and encourages people to comment on the post and rate it, much like a forum.

One of the biggest offenders of the concept of revolutionary is associative databases. I discussed these with one of my coworkers and apparently these "revolutionary" new database systems are in reality the precursors to relational database systems.

In the end, I think I've convinced myself that I will not be creating anything revolutionary in my lifetime. Hell, Hollywood is having trouble coming up with original film ideas and they have an army of writers working for them. Why should I think I'm any better.

More on bazaar

2008-01-17T16:09:00.001-08:00

I just started moving some code from my subversion repository to bazaar and I figured I'd pass on a few tidbits of knowledge.

Converting my old repository didn't work out. I couldn't get any of the tools to work, which may be my own dumb fault.
Bazaar doesn't like large files. I don't know what the limit is, but if I try to add some multi-megabyte files, it will crash. Granted, it does appear to recover just fine if you do a revert, but it's just weird. Subversion appears to handle them just fine. I guess that's one area that subversion does excel at.
Bazaar can be very slow at times. It's happened to me only twice in the past two days where bazaar really slowed down on me. It looked like it was stuck pushing data back to the main branch. It was still running, though I did have to kill it today since I was taking my laptop home.
Repository layout is important. More below.

See, when I started this yesterday, I was going to put one big folder that spanned several projects all in one bazaar repository with the main branch being stored on a shared drive on my company's network. (No need to worry about backups then :) After having to painstakingly add directory after directory, clean out large files, etc. I had a great idea. I was going to create a single repository for each project. This would reduce the amount of space I need since I don't need every project on my local machine. I'll probably keep a couple smaller ones on, but I can just keep the ones I need locally and have the rest archived. Since my subversion repository is still intact and on the shared drive as well, I can just copy over all the directories I need, put them in bazaar and done!

Now, one of the other things I plan on doing is having the code/software and the documentation for a project in the same repository. Now, I have all my documents in my My Documents folder and my code in other folders. I figure this way, I'm less prone to keeping clutter on my drive. Also, I think it'll keep me more organized.

Now, to answer the question of why I'm moving from subversion to bazaar. It's really quite simple: local commits. I really like subversion and think it works great, but local commits are something I want. I know there are other tools, but as I stated in the prior post, bazaar just impressed me with how they think about version control.

Anyway, that's all for now.

How bazaar!

2007-12-16T12:31:00.000-08:00

I've blogged about version control systems in the past, but I never made mention of Bazaar before. Well, I found an interesting link through reddit, I think, and it's worth reading.

http://bazaar-vcs.org/Workflows

Why is this cool? Unlike any other tool I've looked at recently, this is the only one that details the various scenarios a version control tool could be used in. This resolves some of the concerns that I have in that with Bazaar, we can have a centralized server and still be able to version files locally. Best of both worlds!

Needless to say, I'm really going to look into this tool. I figure any project that looks at things like workflows and shows you how to use the tool in those situations as well as ensuring you can use the tool in different scenarios is a good sign. Does this mean that Bazaar is the best solution? Probably not, but it seems like a very good all-around solution.

HTML done right?

2007-11-30T17:17:00.000-08:00

Ugh...it's been a while. I've been tied up with work and have a newborn in the house now. A bouncing, healthy, baby boy! Woo-hoo!

Anyway, HTML. This seems to be a bit of a hot topic in terms of what will be the successor to HTML 4/XHTML 1.x. The two major camps appear to be the W3C with XHTML 2.0 and the WhatWG's HTML 5. Both have good ideas, but both have problems.

I'm not going to go into a debate on which is better because regardless of which one is picked, there are still some fundamental problems with both. My biggest complaint is the mixture of different languages in one document. You can have HTML, CSS, and JavaScript all in one file and mixed together. Not very pretty. Also, things aren't as nice and simple as they could be.

My current thought process is to come up with a different standard that's designed from the beginning to ensure the separation of style, content, and script/code. First, we need to have the document divided up into sections. I see 4 sections: metadata, content, style, and script.

First is the metadata section. This should contain information like the character set encoding used and the version of the document specification used. This can contain more than just information. The two mentioned are just those that are required.

The second is either the content or the style. Not really sure which as I don't know which would be better for a browser to handle first. One could either read in the content first then apply styles as they are read or read in the styles first, then render the content as it's read. For now, I'm going to assume the style comes first because from a browser standpoint, I'm thinking that it would be better to know all of the display information first, then apply it as the content is being read.

The style section is the equivalent to the style section of current HTML document. However, I'm not 100% sure that CSS in it's current state is the way to go. The big problem is that using floats to do page layout isn't the most intuitive way to go. Also, absolute positioning doesn't solve all the problems we encounter. I'm thinking that a good, simple grid-based positioning system would work best. An element would have it's position within the grid specified as a pair of (x,y) coordinates (top left & bottom right). Granted, this may not be the best way to go because I'm not sure how to specify stretchiness. I.e. how the element can stretch at least horizontally depending on the size of the screen.

The third section, content, needs to focus on the parts of a document, such as titles, heading, and paragraphs. HTML is pretty close, but it does have some limitations in that a number of different elements, such as lists or tables, are used for things that they weren't meant for, such as navigation or page layout. Also, I'm not sure what to think about div and span tags. They have no meaning other than one's a block be default and the other is inline by default. Each element should describe the content it contains. This may be a bit of an impossibility because how would you style an individual piece of text that's contained within a paragraph?

Another aspect of the content section that needs to be looked into is the semantic aspect. This is interesting because while it is a good way to try to get everyone to mean the same thing when they say "foo," it's really kind-of a pipe dream because there's no central database or semantic model that defines everything. Without that, how can we be sure that everyone means the same thing? My current thought process is that the semantic aspect is knowing what the different components of the document are, thus we can derived the importance of a piece of text and make searches more accurate. Also, it may be nice to have a method of embedding semantic data in the content itself. Something beyond the basic keywords which would be in the metadata section. They key is that additional semantic data would be specific to the domain of the web site. In other words, the semantic data would be specific to the site. The site would then have a RDF file, OWL file, or a semantic database that represents the data model used. Just a thought.

The last section would have the code that would be run on the page if any is needed. The reason for this being last is that we can be sure that the contents and styles have been loaded and applied first before we attempt to do any document manipulation. This has been a problem with many efforts where the page would still be loading when the browser is trying to execute JavaScript. By making it mandatory that the script be on the bottom, we can ensure that this is not a problem.

Now, for all of these, there's no guarantee that the current syntax or technology is really what we want. My main objection is to the XML-like syntax used by (X)HTML. It's very verbose. A LaTeX-like syntax I think is easier and more compact, however I'm not sure which is better for parsing the document into a document tree that can be manipulated dynamically.

Now, the question some of you may be asking is why I am proposing this. Simple: the web has evolved beyond what HTML was originally designed for, so a new forward-thinking approach is necessary. Also, after using it for many years, we have discovered many problems and we should now take the time to make sure they're not a problem anymore. Of course, while all this is nice and good, the problem is getting it accepted. Browsers are notoriously slow in implementing new standards and many people are used to HTML and don't want to use something different.

What to do...

2007-10-10T17:12:00.000-07:00

Got an interesting idea recently, but I don't know what to code it in. See, you have technology like LaTeX where you can author professional documents using an ASCII text editor. Why not do the same for spreadsheets? Wouldn't it be nice to be able to generate a simple text document and generate the results using a small program?

Yeah, I thought it would be a good idea, especially for scientific applications with large amounts of data. No sense in using a heavyweight program for crunching a ton of numbers.

Anyway, back to the problem: what do I code it in. There are two major portions to this: extracting the data from the data files and performing the actual calculations. My first thought was to use Lua since it's very fast and there is a arbitrary precision math library available for it. However, it isn't the greatest for string manipulation, thus making it a bit harder to extract the data. Perl, on the other hand, is a bit slower overall, however it has great string manipulation capabilities. Specifically, it has a split function, which would be very important for this. It also has an arbitrary precision math library.

Now, to add to the mix a bit, I already know Perl and was going to use this as an excuse to learn Lua, but if it's not the best tool, I probably shouldn't use it.

Hrm...

Who's right when it comes to version control?

2007-08-31T16:28:00.000-07:00

There's been a bit of discussion about who's tools can piss the furthest...I mean, who's tool is the better version control tool. The discussion is mainly between Subversion and Git and which method is right. Subversion relies on a centralized repository. Git is a distributed system.

So, which is better? I say both are good.

Here's my thinking. First, it depends on the organization which fits in better with their business model. A distributed system may not be what the organization wants. Second, they're both good tools, though neither is perfect. Git is better than Subversion for things like branching and merging, however I really like Subversion's diff algorithm and the fact it stores diff's instead of snapshots. Git also doesn't have a hoard of annoying little hidden subdirectories strewn throughout your directory structure. However, Subversion has several good GUI tools that make it much easier for people to use, assuming they're not willing to use the command line. TortoiseSVN is a great tools for Windows users.

I've used both and I like both. Git, I think, is a bit easier to use, but there's nothing wrong with Subversion. In the end, it's what works best for you.

More LaTeX goodness

2007-08-31T16:12:00.000-07:00

Yeah, I'm boring, but I figured I'd pass along some tidbits. Most of my technical goodies I discover or do at work, which really limits what I can blog about. However, this one I can.

You see, I don't like MS Word, just like lord knows how many people. So, when I had to do some documentation recently, like documenting potential requirements or documenting some design ideas, I decided not to use it. They way I figure it, it won't go to a customer, so why should I bother? Besides, I seem to get it done faster in gVim and I want to get some practice for when I really decide to use it.

Overall, I must say it worked well. Images are included very easily. Just needed some tweaking to get the sizes right and all is good. I even got one image to sit on the right-hand side of the document with the text wrapping around it. No fighting or anything. It just works. Sweet!

Now, I did several documents recently using LaTeX, one was an outline that I felt would be better to do on a computer than paper: easier to reorganize. Today however, I was documenting some potential requirements. Nothing big. I figured I'd do it as a table and when done, get it into an RTF Document and finally a Word document. Needless to say, it was pretty damn easy. The only trouble I came across was that the one table I created was pretty long. Turns out there's a package that comes with MiKTeX, and maybe regular LaTeX, called longtable that handles that. Here's a link:

http://www.astro.psu.edu/gradinfo/psuthesis/longtable.html

The only other thing I did was I found a LaTeX class that makes a document look like it was formatted in Word. I can't remember the name of it. I think it was wordstyle, but I can't remember.

The result was pretty good. I had to adjust some margins and the table cell widths. The only real bug was that the LaTeX to RTF tool didn't understand the longtable package, so I ended up having the header displayed twice. Whoopee. I deleted the one row and went on my merry way.

This will be strange to say, but I enjoyed the experience. This either means that I really like working with LaTeX or I'm just that bored at work. Either way, it seems much easier and nicer to work on documents in gVim than MS Word or even OpenOffice.org, which is relatively sane.

Data mining done right.

2007-08-10T15:37:00.000-07:00

This is fantastic. Law enforcement actually doing data mining properly and with good results.

http://www.schneier.com/blog/archives/2007/08/police_data_min.html

What version control systems are missing

2007-08-07T18:37:00.000-07:00

Sorry again for the long delays...meant to blog earlier and then went on vacation.

Anyway, I've been looking into different version control systems lately and I find that there are two things missing. First is a way to make them more reliable. Second is a way to make them perform better when dealing with a central repository.

To address the first, when you have a version control system, you typically have either a central server, like Subversion or CVS, or you have a series of distributed nodes, like git or Bazaar. What happens if the central server or a node goes down irrecoverably? One answer is to go back to a backup, which may not have all of the recent changes.

To address the second, which may not be as big of an issue, there is the possibility that you may have a very busy server. For example, the git repository that stores the entire Linux kernel. Granted, most likely it will be primarily reads, but that can still slow things down.

Replication I think would help both of these, especially for centralized servers or nodes where a lot of changes occur. The small solution would be to use Oracle's BerkeleyDB, which supports replication and is very light weight. Also, though this is less practical for distributed systems, you could build the system on top of a relational database, such as MySQL. Then, you can get both replication and clustering.

The main reason I'm thinking this way is that by using a back-end like BerkeleyDB or MySQL, you don't have to deal with many of the low-level issues. Also, they have the built-in technologies to allow for faster recovery. To me, this can be very important in a corporate or open-source environment.

Food for thought.

Streaming blobs!

2007-07-09T18:15:00.000-07:00

Just a quick posting. I'm really excited about this technology. The ability to stream blobs out of a database really makes things easier in several ways. Granted, this goes against the conventional wisdom that you store blobs outside of the database, but it looks to be real nice for those situations where you have a ton of files. I think it's really going to change things for larger sites where you have lots of images, documents, etc. Instead of having to pull the file out, store it on disk, and send it to the client. Now, we just link to it.

As for the other advantages that putting files in a database, here they are from memory:

1. Overcome filesystem limitations in terms of the number of files that can be stored on disk.
2. Database backups now take care of all of the data, including files.
3. Can take advantage of the benefits of replication and clustering.

Food for thoughts.

Public Key Infrastructure and Key Revocation

2007-06-21T15:49:00.000-07:00

First off, I'm not a security expert, so I may not get this quite right.

Anyway, I was thinking yesterday about key revocation and how you would handle this securely and reliably. It's an interesting problem. Looking at the worst case scenario, which would be a global system, it seems like a clustered solution would work best. We don't want a centralized server because if it goes down, then we can't manage the keys properly. With the cluster, we'll ensure high availability. The problem with this is ensuring that any updates are synced across all servers. Also, we have to ensure that the servers are trusted. This is where the real complexity comes into play as there are many clustering solutions. How we know that a server can be trusted is a big issue and also, if a key is revoked, how do we ensure that the fact the key is revoked is transmitted to all of the servers? Or, do we set up the system to check several servers simultaneously to find the latest updates?

Food for thought...

The Document Markup Conundrum

2007-05-29T18:06:00.000-07:00

There are many languages that exist for formatting documents. HTML is a very common example of this and a good one to look at. It can be used to define the structure of a document and has some rudimentary style options. LaTeX is another markup language that is similar in those respects since it also focuses on structure with some styling capabilities. The problem with this is that you can worry about both the structure and the style at the same time.

CSS is an attempt to fix this problem by providing a means to style a document outside of the document itself, however it's still embeddable and can be declared within the document itself. This can lead to a document that can be confusing and hard to maintain.

The W3C wants the world to move towards XHTML. Others want to move to HTML5. Both have some of the same problems as prior versions, though there are a number of fixes and improvements. In the attempt to retain backwards compatibility, are we really creating the right solution?

I'm thinking we need to start from scratch. To continue with enforcing backwards compatibility is going to hold us back. However, to completely bail on the old formats means we have to wait for a new reader/browser to become compatible with it. Looking at the current and past states of web browsers, waiting for that to happen is probably pointless.

What I would love to see is a standard that is simple to understand and fixes all of the mistakes made with HTML and others. Also, I'd love it to be supported fairly quickly. Some ideas I've had are focusing on structure instead of presentation in the actual document portion itself. One way of doing this is similar to how you can embed data and documentation in a Perl script. After the actual Perl code, you can add a section, such as __DATA__. This becomes interesting for a document as you now have a clear way of having style and script inside a document file without polluting the actual document itself.

Another thought I was toying with is what type of markup should be used. Do we use the XML/HTML style of tags or something like LaTeX where text is enclosed in braces? Each way has it's pros and cons. Neither really appears to have any real benefit over the other from a technical perspective, so it's just a matter of preference or whichever is cleaner. Honestly, the only real difference between the two that I see is that the LaTeX style is more compact whereas the XML style can be more readable.

Now, the more important criteria is which is easier for someone to write a document with. Personally, I found the LaTeX style to be a bit easier, but as a programmer, braces are commonplace. I've also found that it is clearer for me. To say \section{Foo} seems clearer than <h1>Foo</h1>. Outside of a bit of wordiness at times, the only thing that I think LaTeX could do better it tables. They're kind-of a pain to populate, but I do like the way you can customize the column dividers.

Now, as for the styling aspect of things, CSS does a pretty good job, but it does have it's deficiencies. First, I think it can use inheritance, Perl6 is doing exactly what I think CSS needs to do with their regular expressions. In Perl6, you can build a regular expression using other regular expressions. With CSS, it think it would be nice to be able to do the same thing. For example, let's say we have a style for links so that they are all 14 pixels in size, white in color, and have a blue background. Now, let's say I have another style for links that's exactly the same, except I want it italicized. Instead of creating a whole new style that's almost an exact copy of the prior one, we could inherit the prior style and just add the necessary changes.

From a scripting perspective, I don't have many complaints outside of the fact that Javascript is a bit weird at times. However, in terms of DOM access, one new function keeps coming to mind: get ElementsByAttribute. I see it having two parameters: the attribute you want to search on and the value you want to search for. For example, to find all elements with a class attribute of "foo", you could do this: getElementsByAttribute('class', 'foo'); May not be the most efficient way of doing things, it can handle any attributes you may want to search on.

As my last thought, is an organization like the W3C the right way to go about creating and updating a standard? They are not necessarily the fastest at getting updates to specs ready. Look at OpenGL and DirectX. OpenGL is slow to be updated where DirectX is updated much more frequently to take advantage of new technologies. A potential problem with moving faster is that the specification may not be as well thought out as it could be.

Well, that's all for now. If I have the time, I'll try to come up with an actual specification.

Ion Rocks

2007-05-25T15:32:00.000-07:00

As you may have read before, I was trying out Ion as my new WM. It rocks. Outside of one app that didn't want to play nice, it works very well. The best way I can describe it is to say it's like GNU screen on steroids. I can switch between apps, frames, and desktops without a mouse. It's also very stable. It hasn't crashed on me once.

As someone who spends a lot of time on a laptop, this works very well. I don't have a mouse at all times and the touchpad doesn't work very well, so being able to stick with a keyboard for everything is very nice. That's also one of the reasons I like Vim.

It's quite funny. I started out using DOS. Used a number of iterations if MS Windows GUI's as well as a number of Linux WMs and I'm back using something very basic and simple.

Busy...

2007-05-17T18:22:00.000-07:00

Again, bad blogger, but for the past few weeks I've gained an increased hatred for Internet Explorer. I never wanted to talk about work, but I need to vent.

See, we needed to get a demo up and running, so the tech lead made the call to focus on one browser: Firefox. It was simply easier for us to get stuff working Firefox than IE. Stuff got done quick and the demos went well. However, the customer has standardized on IE, so we now have to make sure everything works in IE as well. Based on past experience, I didn't think this would be that hard. Unfortunately, that wasn't true.

You see, before this the most complex Javascript I worked on was an AJAX-like app with form validation and dynamic form creation that was based on a simple design. This, on the other hand, is much more complex with a significant amount of dynamic content. And with the increased complexity are increased headaches.

All I have to say is "My god what the f*** were they thinking!" Here's the first example: if you want to get/set the class name, according to the spec, you use something like "setAttribute('class', 'foo');" Guess what? That won't work in IE. Instead of 'class' you use 'className.' Huh?

Yeah, I've hit a number of bugs that have been found by others, but aren't well documented. I found a couple pretty easily however a few have caused me a couple days of hunting and searching. Needless to say, it's been a trying journey. The good news is that I have squashed or worked around most of the bugs, however it's hard to believe how inane some of them are. Luckily one of the things I've managed to do is create a some functions in libraries to work around some of the issues, like the get/set-Attribute bug.

Well, this experience has convinced me on one thing: if I ever attempt to create a highly dynamic web application, I'm not going to develop for IE. The level of frustration I've experienced is not something I want to deal with at work let alone my personal time.

Lousy blogger...

2007-04-16T19:58:00.000-07:00

Yeah, I suck. I haven't posted in a while, but I've been working on my final Master's class for the last few weeks, so I haven't had much time. However, the good news is that this is providing a nice little testbed for my framework. It still needs some work in the way of bug fixes and I do know it needs a couple extra features, namely caching. After I'm done with my class I plan on adding it. Nothing big. Just a way to capture the output of a page and cache it for a period of time. Not the most advanced thing, but a nice optimization, especially if you're running a slow server or just have a page that takes a while to generate. Again, this will be done in such a way so that it doesn't muck with your coding.

Another feature I am debating is having a couple functions to generate simple insert/update statements. The reason for this is that I've found myself converting insert statements to update statements. I figure that this way if the developer converts a script that does an insert to a script that does an update, they just have to change a function call. Not 100% sure if I want to do it, especially since I just remembered the INSERT...ON DUPLICATE KEY UPDATE statement for MySQL.

Anyway, hopefully this summer I'll be able to update more and, if all goes well, I'll release my first version of my framework. I also have another little project that I plan on doing for fun that I'd like to work on as well. Maybe I'll win a prize...

Framework Progress and New WM

2007-03-21T19:18:00.000-07:00

First, just a quick post on the progress I made with the framework I've been working on. It turned into an interesting exercise in evolution. To start, the index.php page that is used was procedural in nature, but instantiated objects for each page. Nice and easy. Well, I figured I'd try to make the framework a bit cleaner for the developers, so I figured I could use inheritance, so I created a small class with a couple helper functions that the page classes would extend. Well, it worked, but I made a small goof where I goofed the path to the page classes. So, since the variable I needed was outside of the scope of the class, I ended up wrapping all of the code up into the class. So, where I started out with procedural code and ended up with a Object-oriented setup.

Second, I'm trying out Ion as my windows manager on my Linux laptop. So far, I'm liking it. It seems to be very fast and doesn't require me to use the mouse much. Very nice. I've been an Enlightenment fan for quite some time, but I think I may enjoy using this one more.

2007-03-13T19:03:00.000-07:00

Well, since I off-loaded the associative database, I managed to get more work done on another project of mine. Because of the dynamic nature of a project I worked on, and the fact I absolutely hated working with Fusebox, I created a something akin to a framework for PHP. I had three goals in mind when I created it:

Easy to add additional pages in a minimal amount of time.

Low processing overhead.

Don't force the user into a method of development they don't want to use.

To elaborate a bit, to make it easy to add more pages I took advantage of the ability to load classes dynamically. This way I can simply create a new page without having to modify any other files and all I have to do to use it is link to it. Also, the main file, index.php, is very minimal in terms of functionality, so it's very light-weight compared to other.

As for not forcing the user into a method of development, I kind-of lied. I do force you to organize your pages into subdirectories, but outside of that, there's not much that I force you to do. I do have my own way of doing things, but you don't necessarily have to do it the same way. I find that I prefer to build the structure of the HTML first and then insert the content into it. String replace works nice for this. I find that if I get the document structure down first, then I can more easily get a good HTML doc built.

Well, that's all for tonight.

Associative Database...done?

2007-03-09T17:23:00.000-08:00

Well, I decided to stop working on the associative database. I simply don't have the time to give it the proper attention that it needs to go forward, so I put it out into the world on Perlmonks.org. Here's a link for those who are interested: Associative Database

Things software engineers should learn...

2007-03-05T15:13:00.000-08:00

Every once in a while I think of what I know now and what I learned in college and every once in a while I see some discrepancies.

Here are some things that I thought of that I think should be taught as part of a CS course.

Version Control. I don't care what tool is used, just to learn and understand the concepts I think would be valuable.

Basic software engineering. Things like project plans, requirements, design, life cycle models. I'm not looking at anything too in depth. Just enough to give them an understanding of what's going on.

Vi/Vim. I know that there are a lot of people will probably balk at this, however it's a very useful tool and it is one of the few editors/IDE's that exists by default on many different platforms. Even if the students can't use it very well, it's good to at least be able to changes to a file.

Scripting vs. compiled languages. How are they different? When are the best used? How can they be combined? I've personally come to like scripting languages a lot, however I do understand where they're not the best tool for the job.

That's all I have for now. I'm sure I'm short a few things, but I just can't think of what they are right now.

Long time, no post...but progress...

2007-03-04T18:57:00.000-08:00

Sorry for the lack of posts, but I've been busy with school and what not. Anyway, I do have some good news: I think have a tool to create source code documentation. Yes, I know there are others, however I think most are based on Javadoc which I never liked. Granted, I think this is more because people don't know how to document their code.

Anyway, the big difference between what I produced and similar tools I've seen is that what I created exports to LaTeX instead of HTML. The nice part about this is that the documentation can then be exported to several different formats. For example, converting it to HTML allows you to publish it on the web while converting it to PDF makes it easily printable.

I came up with this after learning LaTeX at work to document an API that I am working on at work. It was fantastic to be able to create good documentation that others could use without having to switch programs to create it. With gVim, I split the window vertically and as I finished parts of the API, I documented it. Unfortunately it's a manual process, so I decided to take advantage of LaTeX while automating the process.

The syntax is a bit different from Javadoc. First, I denote blocks of text to be processed with triple brackets. This was done so that only text within these blocks would be processed and nothing else. This can be useful if sections of code are commented out and you don't want them documented or you may have embedded text that's similar to the syntax I used for the documentation options. As for the documentation options, I based them off the LaTeX syntax. This made sense to me because I can define one or more elements for each option. For example, a function description has one element: the description. On the other hand, for function parameters, you could want the name, type, and description of the parameter.

To conclude, I'll probably start using it myself, but if I get a chance, I may put in on sourceforge.net so that others can use it.

Spoke too soon?

2007-02-02T18:10:00.000-08:00

After a bit of thought, I came to the realization I spoke too soon. What came to my head this morning is the question "Why do people want multi-master replication?" This is where the problem with my previous idea comes to play because wouldn't improve insert/update performance whereas real multi-master replication would. At least that's what I'm assuming based on how I understand it working.

So, does this mean that my idea is worthless? No, I don't think so. If you have an application with a high percentage of reads, it may be worthwhile to have each client connect to a replicated server instead of the central server for updates.

Now granted, I don't know the performance characteristics about my previous idea, however I'm going to safely guess that if I have 10 replicated servers and I use federated tables for inserts, I'm can bog down the central server. Not exactly a scenario we want.

Multi-master replication solved with Federated tables?

2007-02-01T20:17:00.000-08:00

As many of you may know, MySQL does not support multi-master replication out of the box. Yes, it is possible to come up with a process to handle it, but they're not always the best or the easiest to implement. Well, today I had a thought: could I simulate multi-master replication using Federated tables?

Here's the idea: we have a central server where all inserts will be applied. We then replicate that server to several other servers. On each of the replicated servers, we put a second database that is strictly for making updates to the data. This second database will consist of only federated tables that are linked to the central server. All inserts, updates, and deletes are applied to the second database while all selects would be performed on the first database, which is the replicated data.

Now, I'm not sure how well this will work, but to me it seems like it would be a fairly elegant solution to the problem. Since all of the inserts will technically be occurring on the central server, we don't have to worry about duplicate keys as much.

I'll have to look into this some more...

LaTeX Goodness

2007-01-11T18:33:00.000-08:00

First off, Happy Belated New Year!

Second, for those of you who haven't tried it, check out LaTeX. After reading a number of articles on why typesetting languages are better than word processors and on why binary blob formats are bad for archival purposes, I decided to try it out. I must say, I really like it. Currently I started working on my resume using it, I'm documenting an API I'm developing at work, and, using the Prosper package, I'm building a presentation. It's really quite simple and it makes nice looking documents. For the API, I'm really loving it because I can document everything while looking at the code at the same time in gVim. Also, it takes care of the title page, page numbering, and table of contents all for me.

Seriously, check it out, especially if your a developer. I'm really loving the fact I can use my favorite editor to make professional looking documents. As I said before, it's especially nice when you're documenting an API while looking at the code.

Computer Security Tips

2006-12-15T19:13:00.000-08:00

First, I'm going to say I'm no computer security expert, so if my advice doesn't provide 100% protection to your computer, all I can say is sorry. Better luck next time.

Anyway, I have been asked on several occasions about making a computer more secure. This is mainly for Windows machines, so I figured I'd do a quick post with a quick listing of the steps I've taken to make my computer more secure.

Switch to Firefox or another browser. I use Firefox. Besides being a better web browser, IMHO, I feel it's less likely to allow an attacker to completely infiltrate your computer since it doesn't use ActiveX nor is it tied into your operating system. If you do get it, I recommend the following addons: Adblock Plus or Adblock, NoScript, and Permit Cookies. These extensions will not only prevent some of the annoyances of web browsing affect you, but you have control over what sites are allowed to execute Javascript and/or set cookies on your machine.

Use a firewall. Currently, since I have a Nforce4 Ultra motherboard, I'm using their onboard firewall, but I've had luck with ZoneAlarm as well. I also have a router with a firewall on it, so the personal firewall is mainly to restrict what apps can hit the Internet. Still, very useful to keep unknown apps from phoning home.

Spyware removal software. I use Spybot S&D and AdAware. I recommend using at least two as they catch different things. I typically run these once a week. You can run them at the same time and they still finish in a reasonable amount of time.

Antivirus. I'm currently using ClamWin, which is a Windows GUI version of ClamAV. I've also used AVG by Grisoft and Norton Antivirus. Can't really tell you if they work or not since it looks like I've never gotten a virus on any of my computers in the past...10+ years? I run this once a week as well. Best to do it overnight, especially since ClamWin is slow. I think it's also a bit more thorough, but I'm not 100% sure.

Use Thunderbird or another email client. I use Thunderbird and I know for a fact it stopped a virus because one was sent, but I couldn't tell it was a virus so I sent it to work to have someone look at it. Needless to say, it was picked up by the company's antivirus software. Anyway, as far as I know, all alternatives to Outlook are not vulnerable to any of the attacks against Outlook. Also, I find the spam blocking in Thunderbird to work very well.

I also avoid using Windows Media player whenever possible and I don't use MS Office at home, but that's because I don't like WMP or MS Office, but I do like free alternatives better. Foobar2000 is a very nice music player and OpenOffice.org works quite well for me.

If this helps, great! If not, let me know.

We got disk storage!

2006-12-01T20:09:00.000-08:00

Just what the title says. It's stupidly simple storage using storable to store the object to disk, but we can at least save and load the data later. Eh, it works.

Anyway, this means the current status is that it's a library that stores a graph/associative database in memory that can be stored to disk at any time. Unfortunately, I can't really stress test it, but I think it's pretty good.

The next step is to put it somewhere. I do have a sourceforge account, so I may put it there. I have several ideas that I plan on releasing, so I'm trying to decide whether to have a separate project for each, or to just have them all in one project. The problem is I'm not sure how far I plan on going with any of them. Some may continue. Others may be a one shot deal. Of course, one compelling reason to have each project separate is that if someone wants to take over an idea of mine, I could more easily transition it.

Eh, thoughts for another day.

KISS, my dear friends...

2006-11-21T19:46:00.000-08:00

Today I was asked to look at a new piece of software that we would be using on a project. No biggie. It is a piece of server software, so I figured it would be "query the server and you will get data."

How wrong I was.

While the demos mostly worked, I could not find out how to access it. I looked at all the documentation and tutorials I found on the web site and I'm quite lost. It was especially difficult because it used some "magic" to handle the requests. Therefore, when I looked at the examples and looked at the URL that was called to make the request, I found nothing in directory that it should be pointing to.

Needless to say, this got me thinking about some of the things I've done in the past year. Most recently, I've been reworking an API to make it easier to reuse. Essentially, there are several libraries with some, but not a lot, of dependencies between each other. One of the problems was that each one was a singleton, therefore there was a good deal of code duplication. I pulled the singleton functionality out into its own library and set it up so you can "register" either an object or just the functions within the object with the singleton. This way if a developer wants a singleton, all they have to do is create a library with the functionality they want and register it with the singleton library. The registration functions have also been designed so that you can in one statement make all the public methods and, if you register the entire object, members accessible through the singleton. Not hard at all.

This experience also got me thinking of several things I came up with some time ago. While I would love to say that I was inspired to create these things, they were really born out of potential necessity. A client I'm dealing with has been known to change their minds on almost a whim, therefore I needed to come up with a way to rapidly build functionality. Therefore, I ended up building three different frameworks, for lack of a better term. All three were very different in what they are meant to do, but they all have one thing in common: it is very easy to add functionality. They are all set up so that you can add functionality without having to modify any existing files by using a plugin-like architecture. Piece of cake.

The final thought I had was "why isn't this software easy to use?" "Why can't I simply write a little bit of code, call it remotely, and see my data?" "Why isn't this EASY?!?!?!" I understand that many APIs, frameworks, architectures, etc. do things for some reason deemed valid by their creators and there are those who love it, but I find many to be convoluted. Why aren't they easier to work with?

Well, as a final thought, I plan on posting some of the code I mentioned earlier. One is an RPC library that I started and another is an architecture for building a web site. I never got to use the RPC library, however the web site architecture is working out very well. Compared to Fusebox, which I have used and hated, I feel this has improved my ability to make web sites quickly. If nothing else, at least I worry less about how my code fits into the architecture and focus on building site. Anyway, I'm going to try to get this stuff onto Sourceforge or something else. Suggestions? The problem is I am pretty busy with a Master's program and I want to try to finish the RPC library and create examples on how to use both of them, so it may be a while. Sorry.

*Apologies for the vagueness at times. I didn't want to accidentally bad mouth a product in the event it may be my own dumb fault that I couldn't get it to work.