|
|
Debian RAID NAS on VIA EPIA HOWTOFrom ThisBlueWiki(Redirected from Datacube install)
How to make a lunchbox-sized, quiet, low-power file server with RAID 1 or RAID 0 This guide will become obsolete with the release of Debian 4.0 'etch'. I've since upgraded my server to a RAID 5 configuration booting from Compact Flash and running Debian etch. For that guide, see Debian 4.0 'etch' RAID NAS on VIA EPIA HOWTO. --MagicHour 02:24, 8 March 2007 (PST)
If you're reading this, you've probably got a file server on your network at home, probably running on an old machine. These installations take up a lot of space, are often noisy and hot, and waste electricity. How about a stable, quiet, cool-running system that can serve files to Macs and Windows PCs, and can either use RAID 1, for maximum reliability when you just can't afford to lose valuable data, or RAID 0, to create huge shares up to a terabyte in size (using 2 500GB drives) at the cost of redundancy? Now you can have it, for under $600 (as of September 2005) for a system with dual 300GB Seagate Barracuda hard drives. Mirrored, that's nearly 300GB of fail-safe storage; Striped, close to 600GB! Best of all, the system draws well under 100 watts (probably closer to 60) and has one, count 'em, ONE quiet fan. The 600MHz VIA CPU draws about 7 watts peak, and doesn't even need a fan. For a system with a single 300GB Barracuda, in which case you'd skip most of these instructions because you don't need RAID, that price drops to about $435 at the time of writing. The beauty of it is that you only have to invest in the case and motherboard bundle once. Just keep swapping out hard drives and reinstalling as your storage needs grow. When EIDE (PATA) dies as a standard, get a SATA card supported by linux, drop it in the single PCI slot, and keep on upgrading year after year. Even better, with the recommended case, this system even looks cool. It's a brushed-aluminum lunchbox sized case with bright, nicely-colored indicator LEDs. Be warned, however, I had to cover the power LED with some tape with black marker on it because it is very bright with a color similar to the green LEDs being used in new traffic signals, and was disturbing my sleep due to where it was placed in my room. Note: this guide cribs heavily from Stan Kaufman's RAID1 on Debian Sarge guide; if you want to build a RAID fileserver but don't want to use the VIA hardware referenced in this guide, you can skip the appropriate sections here and also use his guide for reference. Installation notesThese instructions are based on Debian sarge, which was the 'stable' release of Debian as of September 2005. Given Debian's track record, sarge is likely to remain the stable release for the next couple of years. Debian is the right choice for this installation because it is not bloated and therefore does not waste disk space which you could use to store data, it is extremely stable, and it has excellent package management and security update features. This system will not have a GUI, and you will have to interact with the Linux command line. Once it is installed you can use 'swat' to manage your file shares from a web browser, which is pretty easy. The goal of this system is that you have to mess with it as little as possible once it's set up, so unless you are a truly novice user don't let this dissuade you. Just be aware. Ingredients
Note on hard drivesRemember, the goal here is cool, quiet, and power-saving. Good 5400rpm drives almost always make less noise and less waste heat and use less power than their 7200rpm siblings. You shouldn't be concerned about speed, since the theoretical maximum transfer rate over a 100bT network is 10MB/sec, and modern 5400rpm drives can outdo that in their sleep. Maxtor's Maxline II series is great, but may be going out of production. 7200rpm Seagate Barracuda 7200.8s will also do. Remember, the goal is mimimal noise and wattage. Some sources claim that Seagate has had to disable AAM (Automatic Acoustic Management) on all their drives due to a patent lawsuit, and that the newer Barracudas are noisier than old Seagates when seeking. This would appear true with my 7200.8s, though the seek is still very quiet. Samsung has been consistently cited as a manufacturer of quiet drives, notably the Spinpoint line. Users may be able to activate/deactivate AAM on capable drives using Avoiding problems with RAIDDrives should be of the same make and model, though mismatched drives MAY work as long as the smaller one is the initial install disk. Installing hardwareInstall the hard drives (with jumpers set to master) with the primary hard drive going up top with the mounting brackets and the secondary underneath with the special mounting "hood"-like thing. Connect the temporary 2-device IDE cable to the primary IDE interface (the one closer to the CPU), and one of the round cables to the secondary interface. Connect the 2-device cable to the primary drive and the round cable to the secondary drive; connect power and turn the system on to verify that the BIOS can at least see the drives. If it can't, something is wrong. Set the BIOS boot devices to this order: CDROM, HDD-0, HDD-1. Power the system down, unplug it again and hook the CD-ROM up to the second connector in the 2-device IDE cable, making sure it's jumpered for slave. If you have the power connector splitter, plug it into that, otherwise use the power connector for the secondary hard drive -- this power supply only has 2 drive connectors. You'll have to run it with the hood off and the CD-ROM dangling as all three drives won't fit into the case. Installing the operating systemNote: this section will assume you know to choose the default and/or correct answers for trivial questions like language, keyboard type, etc. Also, you must have a working network connection via Ethernet to the machine in order for this install to work; otherwise, you should install from full Debian binary release CDs.
Configuring mail if your ISP uses Authenticated SMTPYou want to do this in order to be able to receive e-mail when the various monitoring utilities detect something going wrong with your RAID or drives. You only need to do this if you need a username and password to login to your SMTP server before sending mail; see the configuration of your e-mail software to determine if this is the case.
Installing a new kernelIt is highly recommended to install a 2.6.x kernel which is properly configured for this specific hardware and application. Using my custom-built kernelI've built a 2.6.8 kernel which is optimized for the EPIA M 6000E hardware and which contains a variety of other useful options built-in, and a number of things not needed removed. Download the Debian package from the following page and install it using 'dpkg -i': Image:Kernel-image-2.6.8 datacube.20050911.1 i386.deb
Building your own kernelYou need to know what you are doing to do this. I recommend you download my supplied kernel above, install it, and examine the configuration I used, which is present in /proc/config.gz when the kernel is running. If you are not using the EPIA hardware, you will need to build your own kernel; examine Stan Kaufman's RAID1 on Debian Sarge guide.
After installing the new kernelChecking RAID support
Setting up RAIDThis involves several steps. Here's the concept: for each of our original partitions (excluding swap since we're not putting swap into the RAID but rather "striping" the swap -- see below) /dev/hda1 ... /dev/hdan, we'll create RAID1 devices /dev/md0 ... /dev/md(n-1) with the /dev/hdax partition "missing" and the /dev/hdcx present. We'll then copy over the contents of all the /dev/hdax partitions to /dev/hdcx, then boot to the new RAID1 device, add back /dev/hda, and let the RAID1 system rebuild /dev/hda. Thus
Copying the systemAt this point, we mount the new RAID device /dev/md0 to a mount point and copy over the appropriate stuff from /dev/hda. If you're following this guide to the letter, you only need to copy / to /mnt once /dev/md0 is mounted on it, because there's noting in the other big partition which will be for your fileshares.
Setting up the new fstabEdit with your favorite text editor fstab on the new RAID device. Edit /mnt/etc/fstab. NB: the CD-ROM entry has been deleted since it's no longer needed. Note also the swap is left on /dev/hda2 and /dev/hdc2 and is not put in the RAID; the "pri=1" option "stripes" the swap across the two drives. Copy /mnt/etc/fstab to /etc/fstab. Note how / is now /dev/md0. Don't forget to create a directory (in this case /shares) as a mount point for your big partition, and add that partition to the new fstab! # /etc/fstab: static file system information. # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 /dev/md0 / ext3 defaults,errors=remount-ro 0 1 /dev/md1 /shares ext3 defaults 0 2 /dev/hda2 none swap sw,pri=1 0 0 /dev/hdc2 none swap sw,pri=1 0 0 #/dev/hdb /media/cdrom0 iso9660 ro,user,noauto 0 0 #/dev/fd0 /media/floppy0 auto rw,user,noauto 0 0 Setting up GRUBEdit with your favorite text editor /boot/grub/menu.lst on the new RAID device, copy it to the first, and install grub on the second drive. # add these entries at top of boot list title Debian GNU/Linux, kernel 2.6.8 RAID root (hd0,0) kernel (hd0,0)/vmlinuz ro root=/dev/md0 md=0,/dev/hda1,/dev/hdc1 savedefault boot title Debian GNU/Linux, kernel 2.6.8 RAID Mirror Recovery root (hd1,0) kernel (hd1,0)/vmlinuz ro root=/dev/md0 md=0,/dev/hdc1 savedefault boot Edit /mnt/boot/grub/device.map and add a line (hd1) /dev/hdc, then copy that file to /boot/grub/device.map. Now run grub> device (hd0) /dev/hdc grub> root (hd0,0) grub> setup (hd0) grub> quit What this does is make grub think that either /dev/hda or /dev/hdc is equivalent to (hd0), the first hard drive the BIOS finds during boot. In other words, this means that grub can boot from either drive when the other is out. Fixing partitions on first driveAgain, use 'cfdisk' on /dev/hda and change all non-swap partitions to type 'fd', RAID autodetect. Then Write and Quit. RebootingReboot the machine and if all is well, it will reboot with root running on the degraded RAID device -- remember, the first disk is still 'missing.' Create /shares and mount the big RAID partition to it. /proc/mdstat will tell us that the first disks in each RAID volume are still missing. Use mdadm to add them and keep looking at /proc/mdstat to check the progress. Note that a partition of close to 300GB on this machine will take well over an hour to sync. While that big volume is syncing, you can get rid of a lot of unneeded packages to make your Debian install leaner, faster booting, and more secure. See below. # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1]
md1 : active raid1 hdc3[1]
291017408 blocks [2/1] [_U]
md0 : active raid1 hdc1[1]
1951744 blocks [2/1] [_U]
unused devices: <none>
# mdadm --add /dev/md0 /dev/hda1 mdadm: hot added /dev/hda1 # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1]
md1 : active raid1 hdc3[1]
291017408 blocks [2/1] [_U]
md0 : active raid1 hda1[2] hdc1[1]
1951744 blocks [2/1] [_U]
[===>.................] recovery = 18.5% (361600/1951744) finish=0.4min speed=60266K/sec
unused devices: <none>
Trimming the installationUnless you are talking to ancient Macs or *nix machines which use NFS, you can safely remove NFS and netatalk, along with some other packages which take up memory and may be unwarranted security risks if you're not using them.
Do not remove nfs-common or portmap if you plan on using autofs (automount). This is particularly useful if you want to set up a mount point in your shares folder and have a USB drive automatically mounted to it when it's plugged in; see below for further details. Installing useful utilitiesNTPIt's a good idea to keep your clock synchronized to the appropriate time on a file server where timestamps can be important, and the best way to do that is NTP. Installing ntp, ntp-simple and ntp-server should work out of the box to keep your clock synced reasonably closely to the real accurate time, using pool.ntp.org. Additionally, you can now sync your other computers to this machine using NTP (Date and Time control panel, Internet Time tab in Windows XP) -- quite helpful when you want timestamps to match across the various machines that may be storing files on the server. smartmontoolsSmartmontools is a package which gives access to S.M.A.R.T. data from compatible hard drives, and which can monitor drives for signs of impending failure. Install the smartmontools package with aptitude or apt-get. Edit /etc/default/smartmontools and uncomment the line that starts smartd. Then put something like the following in /etc/smartd.conf: /dev/hda -a -o on -S on -n standby -H -l error -l selftest -s (S/../.././04|L/../../5/06) -m youremail@domain.com /dev/hdc -a -o on -S on -n standby -H -l error -l selftest -s (S/../.././04|L/../../5/06) -m youremail@domain.com This enables various types of S.M.A.R.T. monitoring and will send you warning e-mails if something bad is detected, plus the -s option enables a short self-test every day between 4-5am and a long self-test every Friday between 6-7am. 'man smartd.conf' for details. To test that the e-mail will actually work, append -M test to the above lines and start smartd with '/etc/init.d/smartmontools restart'. Then check your e-mail in a minute or two. After you're satisfied that it works, remove the -M test and restart smartmontools. Note that the self-tests degrade performance while they are running, so plan their timing carefully if you enable them. Some S.M.A.R.T. data can be collected without doing self tests, but it's good to do one at least once in a while. Configuring mail for monitoringWhen mdadm was installed, Debian helpfully put a script in /etc/init.d/ to start it up in monitoring mode for you. This script will send mail to root if something goes wrong. If you didn't tell Exim4 to send mail to an outside e-mail about, and took the default of "real-[username]" during setup, you want to change that now. Edit /etc/aliases and replace the root: real-username line with root: youremail@domain.com. Then you need to test that it's working. Do 'mail youremail@domain.com' from the command line (a single period on a blank line ends the message) and 'mail root'. Both messages should show up at youremail@domain.com, which means that not only will mdadm mail to root about RAID problems get through, but smartmontools directly to the e-mail address you put in the config file will get through. Setting up the networkYou may wish to give your server a static IP on your LAN so that you can ssh to it or access it via IP in other ways should the need arise. You can accomplish this by editing /etc/network/interfaces or possibly by giving the machine a static DHCP address on your home router if you have one. Testing the RAIDWe installed grub into both disks so we should be able to boot with either disk. To test this, shutdown and power off the computer, unplug the power to one of the hard drives, then restart the computer. The computer should boot from the remaining disk. It will reboot unless you have a brain-dead BIOS which detects the missing hard drive and waits for the user to press F1 to continue or F2 to enter BIOS setup. However, the good news is that with an adequate UPS and new drives, the likelihood of simultaneous drive failure and reboot (usually from a power outage) is remote. In any case, once the computer has rebooted, you'll see from /proc/mdstat that the one drive is missing. Shut down and power off the computer again, then reconnect the drive's power. Reboot and now you can add the missing drive back in with mdadm. Make certain to allow the drive volumes to re-synch completely before you do anything else (may not be true). You then can repeat the process with the other drive. Setting up SambaThis portion of the guide covers only common tasks in setting up file shares in a home or small-office environment; users needing more complicated capabilities such as groups and varying permissions should make use of the extensive help links available in swat, and read the Samba documentation in /usr/share/doc/samba-doc and /usr/share/doc/samba, as well as online (for Samba 3). Note that the default config currently won't work with browsing the Network share in OS X -- it hangs Finder. Use command-K and enter smb://server.address.here/sharename to access shares (fix coming up). Getting startedMake some directories in /shares for the various file shares you want. Then edit /etc/inetd.conf and uncomment the line for 'swat'. You should now be able to connect to the machine with a web browser on port 901 at http://x.x.x.x:901/ where x.x.x.x is the machine's IP. Log in as root when it prompts you. It is not advisable to use swat over an unsecured wireless connection, as anyone could potentially sniff your root password when you log in. Globals
PrintersUnless you intend on sharing a printer from the server, which is quite possible but currently beyond to scope of this document, you may wish to disable the printer shares.
SharesHome directoriesUnless you know what you are doing, you should choose the Open sharesTo set up an open share which anyone on your network can connect to regardless of NETBIOS username (generally their login name on Windows XP and MacOS X), you want to map every connecting user to the guest user, and map users with bad login info to the guest user, and allow the gust user to read/write the share. Clearly, if security is an issue, this is not a good idea as anyone on your network could delete your entire share. See the other types of shares for more information. If you're in a trusted environment such as a home network behind a broadband firewall, this is the easiest type of share to use. These instructions will cause all files to be written to the share as user:group nobody:nogroup (the default guest user) on the file server.
Please note that if you use the same username on any of your client machines as the non-root username you created when installing Debian, you will need to set a Samba password on the server for that account. Log in as the user you created, run 'smbpasswd' and enter the password you intend to connect with. You'll need to change this by hand by logging into the server and re-running 'smbpasswd' if you want to change it. Alternatively, you can set User-access sharesThis section describes how to set up shares available to users with accounts on the server and valid passwords in the smbpasswd database. You will have to create accounts on the server ('man adduser' as root) and then give them Samba passwds ('smbpasswd user' as root); they can then use this username/password combination to access these shares. You can use the
After setupOnce your shares are all set up and you've verified that they're working the way you want, it's a good idea to disable swat by commenting it out in /etc/inetd.conf until you need to change something. Like any server software, swat is a potential security hole, so it shouldn't be left running when it's not needed. Adding automount support for USB drivesThis section is largely taken from here. If your kernel is configged right or you're using the provided kernel, apt-get install autofs. (section incomplete) Other uses for your systemCVSYou can use CVS, or Concurrent Versions System, to use your system as a reliable place to store source code for cooperative development. The CPU and memory requirements of CVS are quite lean, so your EPIA-based system will be more than adequate for most projects. This section does not explain the use of CVS pserver, and is merely an overview of what is possible; interested users without CVS experience should read the documentation.
(section incomplete)
Squeezebox2 SlimServerIf you're storing your MP3 files on your server and you own a SliMP3, Squeezebox or Squeezebox2 from http://www.slimdevices.com, you can install the SlimServer software directly to your server and use it to stream music to your player, run a web interface to control your player and manage playlists, and handle Internet radio for your player. Since your server will likely be always on, it's a great candidate for running SlimServer instead of any client machines on your network. Even if you don't own one of the hardware players, SlimServer can still be used to stream your music to remote locations (such as your computer at work) from your server via a broadband connection. To do this, you must forward a port on your broadband router, usually port 9000. Please note that the SlimServer cannot play files purchased from Apple's iTunes Music Store due to Digital Restrictions Management (DRM). The slower EPIA machines (such as my 600MHz model) are rather slow for running SlimServer, which is a giant resource-intensive application written in Perl. Your Squeezebox player may act sluggish at times. As the player is a thin client for the server, it's the server's fault. I imagine a 1GHz or better EPIA would perform far more quickly. The trick is to be patient, especially when selecting large playlists or groups of songs to play, or when searching for music. If you start hitting buttons on the remote when you just told the player to play a directory with 500 songs, chances are it might abort before populating the playlist. Then your playlist will be incomplete. So chill! As of summer 2006, SlimDevices has a beta Debian package of SlimServer 6.5b1 available. If you wish to see the obsolete installation instructions for 6.2.1, check the history for this article here. The transcoding instructions may still work, so far I haven't tested this with 6.5b1. SlimServer 6.5b1
TranscodingIf you want to transcode certain formats for streaming remotely or to save bandwidth streaming to a Squeezebox (by encoding WAV to FLAC on the fly, for example) it's easy to install codecs at least for MP3 and FLAC. I have not attempted Shorten, Monkey's Audio, AAC/Apple Lossless, or WMA (with wmadec). SlimServer 6.2 builds include a firmware upgrade that allows the Squeezebox2 to play WMA natively. In a test of transcoding FLAC to MP3 on the fly and then streaming it to a remote Winamp instance, the 600MHz VIA C3 was easily able to handle the job at a sustained load of 27-30% with Player Settings set to a qval of 9 (poor quality but fast) and a bitrate of 320Kbps. Setting the qval to 5 and the bitrate to 192Kbps resulted in a sustained CPU load of 64-71% with spikes to 100% during prebuffering. Preliminary tests revealed no skipping or stuttering in the stream.
See alsoExternal linksTODO |