Thursday, September 6, 2007

Build your own Linux/Asterisk/dhcpd/bind/whatever-you-want "Appliance"

The whole concept of the linux appliance has been interesting to watch evolve. Open source software has led to a huge array of devices that companies may not have had the resource to develop, were it not for freely available source code for operating systems, network stacks and a variety of other things.

I've been searching for appliance hardware for some time now to help me with needs we have for providing services to branch offices. We're trying to cut our technology start up costs and reduce the amount of bandwidth we need to provide to these locations. We've tackled the bandwidth problem by switching to a terminal service based model. Not only has this allowed us to lower bandwidth substantially, but we can also purchase thin client hardware which is cheaper, has a longer duty cycle than a pc and is more secure since no files can be stored on the local client.

The next challenge is the architecture hardware at each branch office. The office's need to be somewhat survivable. If the WAN connection goes down the phones, pc's and thin client stations need to be able to operate. This means dns, dhcp and VoIP need to remain active. In the past we would do this with a VoIP server and a Windows Domain controller. The solution works, but it requires a bigger UPS as well as two servers which generate heat, have moving parts that can fail and are expensive. If a branch office only has 10-30 people in it and all are using TS do we really need that kind of horsepower?

So, I've been looking for a solid state appliance that I could use to provide backup services at each of these locations. I have a few requirements that may or may not apply to other people but they are important in the environment I'm in. The device had to have no fans or disks. It had to be something mass produced (I can't order a run of custom appliances). It should be able to run a standard linux distribution (no need to retrain employees on custom distros). It should be relatively inexpensive. Also, it would be nice (not a requirement) if it had an expansion slot.

Now a lot of people hack Linksys routers and other off the shelf equipment like that. I think it is great and a wonderful application, but it would never be taken seriously as a solution in the corporate environment I work in.

I've searched the Internet off and on for months with no luck finding what I was looking for. Then one day I was walking through my Tech Support office and it hit me. The very same HP thin client devices we were deploying for our end users may meet all of my requirements. They are available from the factory with Debian installed so Linux should be possible and they have an optional pci slot if we needed expansion. Perhaps this was the answer I was looking for. I Googled around to try to find a starting point, surely someone has done this before me, but I found nothing. If you know of any links please post them as a resource for others who may find this article!

So, to document for myself (I'm forgetful at times) I've posted my instructions here. Hopefully they can help other people too. I'm no Linux guru but I can pretty much figure out whatever I put my mind (and time) in to. I'm hoping that by putting this out here other people who are more knowledgeable than myself will collaborate and improve this platform.

One last note. Remember, this is to run a branch office or a small business, not a multi building campus. If you want to run a substantial amount of phones on one of these you should look into OpenSER or SER. I've installed both on these devices and it works fine.


I'm using what we had in stock at work, but there are a variety of models. My philosophy is that an extra $100 for more features is cheaper than struggling through trying to make a less standard installation work on less capable hardware. Like I said, I'm no Linux guru so it is just plain easier to spend $100 rather than roll my own distro of Linux.

I have two thin client devices that are both made by Hewlett Packard, model t5720. One has 512 MB of flash and 256 MB of RAM while the other has 1 GB of flash, 512 MB RAM and the add on PCI slot riser. Both have a pci slot but it requires a riser kit that includes a wider case cover to make room for a card (about $35). Both machines only have a flash drive rather than a spinning hard drive and there is a "radiator" that runs around the unit to negate the need for a spinning fan. Simply put, the devices are solid state with no moving parts, 6 usb ports, audio in/out, ps2 for keyboard and mouse, parallel port, serial port, ethernet port and a video port.

Without riser card extension the case is roughly 4.5"x9.5"x10.5"

You can buy them wherever you like. I like CDW because they have a good selection of accessories for them and they are one of our preferred vendors at work anyway. They are available with XP embedded or Debian Linux. The ones with XP have more available RAM and Flash. If you can't stand giving Bill another dollar you could buy a Debian one and upgrade the RAM and Flash, but it will be more expensive in the long run.

Operating System

I've always used Red Hat based distros like RHEL, CentOS or Fedora. The problem with these distributions became apparent to me when I started installing them on the t5720 that only has 512 MB of Flash. They're very bloated, even if you turn off all options you can't get Fedora 7 to install. I was going to have to venture into the unknown, but I really wanted to keep a mainline distribution. Damn Small Linux ran great, but there isn't an easy way to install the additional software I needed. I think for the future getting DSL to run on these with some server apps like Asterisk would really be killer, but I don't have time to learn all of that right now.

I finally ended up settling on Debian Etch. Oh boy, another packaging system and file structure to learn (heavy sarcasm). Funny thing is, once I got to using it I really liked it. Here are the steps I took.

1. Download the Debian Etch netinst CD image. At the time of this posting it was located here. The netinst CD will install the basics to get Etch up and running. Since we are pressed for disk space, this is the best option. Gotta leave luxuries behind. Truth be told, even the netinst install is a bit bloated for our application but it is a standard and well known Linux distro so it should keep the learning curve down.

2. Burn the .iso image to a CD. Then scare up a usb CD-ROM drive that you can plug in to the t5720. If you know how to build a bootable usb thumb drive then you could go that route instead, but that left me nothing but frustration. You could also do a pxe network install if you know how.

3. Plug in your usb CD-ROM drive and put the Debian netinst CD in. Turn on the t5720. When the POST screen comes up, hit F10 to go into bios setup. Go into "Advanced CMOS features" and make sure that the first boot device is USB. Then go back to the main menu and choose "Power Management Setup". Set the PWRON After PWR-Fail to "On". If there is a power failure longer than UPS backup can handle at the branch office (or wherever you deploy your appliance) we want the appliance to boot once power is restored. Now hit F10 to save your preferences and exit.

4. I'm assuming some skill level here so I won't go into terrible detail. When the Debian startup screen comes up just hit Enter to boot. You will then be asked for your language and territory settings, so enter what is appropriate. Setup your hostname and domain. When it comes to disk partitioning you will have to do it manually if you only have a 512 MB flash drive. If you have the 1 GB Flash drive you can accept the defaults. From here on out you can do whatever you like until you come to the "Configure the package manager" screen.

5. At the "Configure the package manager" choose to use a mirror. I suggest you use the mirror because I got errors about the architecture not being supported at other mirrors. I choose no on the package survey option, mostly just to keep as many bytes of Flash free as possible (every little bit helps). At the software selection screen I choose absolutely nothing (uncheck "Standard system"). If I need something I'll install it later. Again, keep as many bytes of Flash free as we can.

6. Let grub install on the master boot record so Debian can, well, boot. After that the CD should eject, the appliance will reboot.

7. To finish the OS install I have to do a couple things. First, assign a static ip address. This can be done in /etc/network/interfaces. Again, I assume some Linux experience so I assume you know how to use a text editor. I changed this:

iface eth0 inet dhcp

to this:

iface eth0 inet static

Obviously you need to use the appropriate ip settings for your network. All of the documentation I have read says you can just do a "/etc/init.d/networking restart" at this point, but my experience is that eth0 doesn't release the dhcp ip address. It's easy enough to fix with a reboot so I just do that once I'm ready.

8. Since you are now off of DHCP make sure your dns settings are right. They are stored in /etc/resolv.conf. Mine looks like this:


9. The next thing I do is remove the CD ROM as a source for package installation. If I have this appliance 400 miles away the last thing I want to do is need a CD-ROM drive plugged in and have the netinst CD on hand. This is done in /etc/apt/sources.list. I simply comment out the line that starts with "deb cdrom:" by putting a # in front of it. Then I run apt-get update to save the new repository settings.

10. Now, I install the openssh-server package with "apt-get install openssh-server" since it isn't part of the netinst CD install. Once this is done I perform the rest of the configuration from PuTTy (remember, we're mostly a Windows shop).

11. The last step is that I run a "df -h" to see how much hard drive space I have. Initially I have 523 MB left. After running "apt-get clean" I have 563 MB. This is quite enough to run a couple network services on and then some.

You now have a working Debian Etch server on a solid state "appliance". Heck, it's even branded with the HP name so people who insist on no whitebox servers can't even argue with you ;-) Now you just have to decide which services you want it to host. I'll be installing Asterisk, dhcpd and bind in future posts. You can pick and choose from what I have done or go in a totally different direction. If you need additional hardware you can insert a PCI card or plug in a USB device. You can view this post to see how I installed Asterisk on the t5720.


  1. Have you made any config changes to alleviate stress on the flash storage? I haven't seen specific specs on the module used in the t5720, but in general I've read that flash lasts only about 100K writes.

  2. I was wondering the same thing. Can you please post an update on this (important) issue?

  3. I attended the embedded Asterisk seminar at Astricon this year and from what I understand the way that flash writes are balanced across the "drive" this is no longer as big of a concern as it once was.

    That said, there were suggestions on how to limit the amount of writes to the RAM if you were concerned about such things, but I didn't take detailed notes and can't find the presentation for the life of me. Suggestions were about different types of filesystems and using RAM as the filesystem while running and then writing the changes to flash on shutdown. The general consensus I got from the presentation and from those in the audience was that this isn't a concern anymore. As with anything, your mileage may vary.

  4. Being pressed for space, and wanting a easy to use distro (as long as you are not afraid of modifying text files) check out Arch Linux. I have a Thinkpad with a Core2 2.2 and I run arch/fluxbox. I just love that when I am not doing anything my cpu usage is 0. It is more of an advanced distro, and it has up-to-date packages (not the best for a server enviroment) but it works great, and it is a small install.

  5. I am not sure of this build, but if you are going to be running this appliance off of flash, you may be able to lower the ram that the OS sees, and then use your remaining ram for your swap.

    I used to do that with pico-bsd booting off of floppies. It would give 2 MB to the OS, and use the remaining 6mb for swap.

    You could do the same thing here, but replace MB with GB ;)