How to replace a disk in RAID6

On one of the server, a disk had some problems so it was time to change it.
Machine is Ubuntu 10.04 with 4 disks in RAID 6 and using LVM.

First, let's "remove" the failing disk (sdb2) from the RAID:

$ mdadm --manage /dev/md0 --fail /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md0

$ mdadm --manage /dev/md0 --remove /dev/sdb2
mdadm: hot removed /dev/sdb2 from /dev/md0

Now if you run lsblk you can see that the disk doesn't belong to the RAID.

-> shutdown the machine
-> replace the disk
-> start the machine

When the machine started, there was a problem with grub and grub rescue command prompt showed up.

To avoid to mess with grub, I decided to use Ubuntu Desktop live CD. I download it on a USB stick and boot the machine on it.

I had the following issue:

missing parameter in configuration file. keyword path

To fix it, simply type live and press enter.
Note: if you hit tab, you can see a list of options.

Perfect, it is working. Now let's install some packages:

$ sudo apt-get install mdadm lvm2

Let's create the partition of the new disk (didn't work for me):

$ parted -a optimal /dev/sdb
(parted) mklabel gpt
(parted) mkpart primary 1 2
(parted) set 1 bios_grub on
(parted) mkpart primary 2 -1
(parted) set 2 raid on
(parted) print

With this method, there was a small unallocated partition at the end... So I couldn't add the disk to the RAID because the size of sdb2 was too small.

Instead I used dd.
First you need to calculate the count parameter.
count = (128*N)+1024
Where N is the number of partitions you have. In this case I had 2, so the result is 1280.

The following commands will copy the partition table from /dev/sda to /dev/sdb. Make sure you type the second command correctly!

$ dd if=/dev/sda of=GPT_TABLE bs=1 count=1280
$ dd if=GPT_TABLE of=/dev/sdb bs=1 count=1280

Then I had to reboot the machine.

Finally add /dev/sdb2 to the RAID:

$ mdadm --manage /dev/md0 --add /dev/sdb2
mdadm: added /dev/sdb2

Then it will take time to repair everything. You can see the process using this command:

$ cat /proc/mdstat

My original disks are Seagate and I tried to add a Western Digital. Unfortunately the size was different by 1M, so I couldn't use the WD disk. I ordered a similar Seagate disk, and the size was OK.
The disk couldn't be added because the size was smaller... and by not much. For example(fdisk -l):

Seagate Barracuda /dev/sda 2,000,396,746,752 bytes
Western Digital   /dev/sdb 2,000,395,698,176 bytes

The best solution is : never us the full disk! Leave a 10 or 15M at the end. So disks with small differences in size can be added.

Add node to Puppet

In this post, I will explain how to add a node (computer) to Puppet.

I assume you have a Puppet master server running and it can ping the future node agent.

Puppet master server:

• Debian 6.0
• Name: master-puppet
• Puppet version: 3.7.4

Future puppet node agent:

• Ubuntu server 14.04 TLS
• Name: new-node

NOTE: Puppet version 4.0 will be out soon, and the process may be different.

On the AGENT

Download puppet:

wget --no-check-certificate https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

Note: if you use a different OS, please download the appropriate file

Install Puppet:

dpkg -i puppetlabs-release-trusty.deb
apt-get update
apt-get install puppet

Make Puppet to start on boot. Edit the file /etc/default/puppet:
START=yes

Run Puppet:

puppet agent --no-daemonize --onetime --test
Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
Info: Creating a new SSL key for new-node
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for new-node
Info: Certificate Request fingerprint (SHA256): 4A:...:04
Info: Caching certificate for ca
Exiting; no certificate found and waitforcert is disabled

On the MASTER

Sign the certificate created by the node agent:

puppet cert sign new-node
Notice: Signed certificate request for new-node
Notice: Removing file Puppet::SSL::CertificateRequest new-node at '/var/lib/puppet/ssl/ca/requests/new-node.pem'

Add the node info in the file site.pp of Puppet.

on the NODE

Run puppet:

puppet agent --no-daemonize --onetime --test

And that's it!

reducing the advertised EDNS UDP packet

I had to set up a PPPoE connection on my Ubuntu server. With the help of the command pppoeconf, it is kind of straigtforward.

However, when browsing some Websites I could see in the log file:

... after reducing the advertised EDNS UDP packet size to 512 octets

And even worse, some Websites were not showing up (ex: finance.yahoo.co.jp).

Solution

There were 2 problems:

1) Problem with IPv6
I had to disable IPv6 (probably a problem with my ISP?) with lines in /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1 

Then run:

sysctl -p /etc/sysctl.conf

2) Problem with packet size
A rule for iptables was automatically created in /etc/ppp/if-up.d/0clampmss. And later on I was flushing all the rules to set my own rules (set in another file). Because I was not re-running 0clampsmss then the rule was not added.
And this rule was for resizing the packet.
So in my own rules, I stopped flushing the table mangle.

And that's it.

How to mount a single disk that was part of RAID 1

I screwed up and deleted the folder /etc of our main computer (samba server, dhcp, router, etc.). Terrible mistake...

The positive thing is that it was an old Ubuntu 8.04 and we wanted to update it.
So here was the time to do it.

On the machine, there was 2 hard drives in RAID 1 (mirror). So we removed them and put 2 new drives. We installed Ubuntu 14.04 and all the necessary packages.

Then it was time to copy data from the old disks to the new ones.
I didn't want to lose any more data, so I wanted to connect only 1 hard drive.

The problem is that this hard drive was part of a RAID, it is not a normal hard drive that we can mount as usual.

If you try, you will get error like:

mount: unknown filesystem type 'linux_raid_member'
mount: /dev/sdc2 already mounted or /mnt/recovery busy

Solution

Here are the steps to do:

1)  create a mount point

mkdir /mnt/recovery

2) list the disk information

fdisk -l

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000abbde

Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *          32    62500863    31250416   fd  Linux raid autodetect
/dev/sdc2      1937904885  1953520064     7807590   fd  Linux raid autodetect
/dev/sdc3        62500864  1937899519   937699328   fd  Linux raid autodetect

I want to mount /dev/sdc3. The important information that we need in the next step:

• sectors: 512
• start: 62500864

3) finally run these commands:

$ losetup --find --show --read-only --offset $((62500864*512)) /dev/sdc
/dev/loop3

$ fsck.ext3 -n -v /dev/loop3
e2fsck 1.42.9 (4-Feb-2014)
Warning: skipping journal recovery because doing a read-only filesystem check.

$ file -s /dev/loop3
/dev/loop2: Linux rev 1.0 ext3 filesystem data, UUID=dbf4-...-9a98a (needs journal recovery) (large files)

$ mount -t ext3 -o ro,noload /dev/loop3 /mnt/recovery

note: I'm using loop3, but it can be a different number in your case.

And that's it! you can now access your files.


If you have an error like:

mount: wrong fs type, bad option, bad superblock on ....

It probably means you have entered a wrong offset.

In case of error, use the following command to see the log:

dmesg | tail

Another solution on a computer without any RAID

On my desktop, I have no RAID disk, so I could run the following command:

mdadm --examine --scan

And that's it!


Here are 2 links that helped me a lot to find the solution!
http://digital-forensics.sans.org/blog/2011/06/14/digital-forensics-mounting-dirty-ext4-filesystems
http://unix.stackexchange.com/questions/72279/how-do-i-recover-files-from-a-single-degraded-mdadm-raid1-drive-not-enough-to

How to send an email after a push to gitolite

This is for the gitolite package on Debian.

# cat /etc/gitolite/VERSION
1.5.4-2+squeeze1 (Debian)

I assume gitolite is working correctly. You can commit, push and pull.

Log in as the gitolite user:

su - gitolite

In my case $HOME is /var/lib/gitolite

Go to the folder $HOME/.gitolite/hooks/common

If the file post-receive exists, then create a copy somewhere else.

cp post-receive /tmp/post-receive

Copy the hook to send an email and rename it:

cp /usr/share/gitolite/hooks/common/post-receive-email post-receive

Make sure privileges are OK:

chowm gitolite:gitolite post-receive
chmod 755 post-receive

Edit $HOME/.gitolite.rc

# edit this line:
$GL_GITCONFIG_KEYS = "";
# for this:
$GL_GITCONFIG_KEYS = ".*";

Then run this to apply the hook to all the repositories:

gl-setup

Now look at one of your repositories. In my case:

ls $HOME/repositories/my_repo.git/hooks

Then you should see the file post-receive.

Finally edit $HOME/.gitolite/conf/gitolite.conf

repo my_repo
    RW+  = user1
    config hooks.mailinglist = destination@test.com
    config hooks.emailprefix = "[my_repo] "
    config hooks.envelopesender = from@test.com
 

Now you can push something to my_repo and should receive an email. Make sure you have setup
an email server correctly.

if you see this message when you push something:

remote: sed: can't read ./description: No such file or directory

It is just a warning. You have successfully pushed your code, but in the email you received, you will see:

It was generated because a ref change was pushed to the repository containing
the project "".

See the empty double quote at the end?

If you want to fix this and the warning message, simply create a file:

vi $HOME/respositorirs/my_repo/description

And inside the file write something, for example:

my_repo

Then if everything is good, the email will look like:

It was generated because a ref change was pushed to the repository containing
the project "my_repo".

Raspberry Pi for beginner - Part 3a: Wifi firmware issue

We don't live in a perfect world.
The steps in Part 3 didn't work for me because of my WiFi dongle (Buffalo WLI-UC-GNM2). If it worked for you, then fine, you can skip this part.

What was the problem?

After completing Part 3, I inserted the micro SD card in the Pi and turned it on. After few seconds, I could see this message:

Wait 8 seconds for Network Interface to come up

And after that a message saying it was impossible to reach the server.

For some person, the solution was to change the power supply (USB cable) for a better one.
But in my case, the problem was the WiFi dongle!

How to solve it?

Here you have 2 cases, please choose:
1) You can't connect the Pi to your router with a cable
2) You can connect the Pi to your router with a cable

1) You can't connect the Pi to your router with a cable 
You can't use the installer. Instead, you have to download the Standalone image.
You also need to download usbit.

• extract the files somewhere on your machine
• insert your memory card in your computer
• open usbit 
• select your memory card
• click on restore
• select the image sd.img

Wait a little bit and that's it! You have a working copy of Raspbmc! But it's not end of our journey!

2) You can connect the Pi to your router with a cable
Same thing as in Part 3, except in the network management popup, don't select Wireless and set the correct values to connect to your router.

Either you chose 1 or 2, here is what you should do now.
Insert your micro SD card in the PI (be careful in which direction!) and plug your Pi!

After a while, you will see XBMC. First, we will disable CEC Adapter, otherwise we won't be able to run the terminal. CEC adapter controls your TV. If you keep it enable, when you exit Raspbmc, it will turn off your TV. And when you turn on your TV, it will restart your Pi. You can re-enable it later.

• go to Settings 
• go to System 
• go to Input Devices 
• go to Peripherals
• open CEC Adapter 
• uncheck the box Enabled 
• click OK 
• go back to the home screen (press Esc key)
• click the standby icon at the bottom-left
• select Exit 
• at the blue splash screen which says Press ESC for command line, press the Esc key

Nice job! You are now ready to access the terminal! Let's continue. At the login prompt:

• enter username pi
• enter password raspberry

Only for the first time you use the command line you have to setup the locales and the timezone. Select the appropriate values. Then you should see:

All Set up now! Thanks for using Raspbmc

WiFi dongle setup

Now let's find out why we can't use our WiFi dongle.
Let's check which chipset the Buffalo WLI-UC-GNM2 is using. Type the command:

lsusb

And the result is:

Bus 001 Device 004: ID 0411:01ee BUFFALO INC. (formerly MelCo., Inc.) WLI-UC-GNM2 Wireless LAN Adapter [Ralink RT3070]  

Now open this page: WiFi
You can see a list of chipsets and if they are directly compatible with Raspbmc or not.

If you look at the WiFi page, you can see RT3070 is not compatible by default. You have to update the firmware. Fortunately there is a help page rt2800usb.

Here again there are 2 cases:
1) You can't connect the Pi to your router with a cable
2) You can connect the Pi to your router with a cable


1) You can't connect the Pi to your router with a cable
I think the best solution is to download on a USB stick the required package(s) using another computer.
Plug the stick in the Pi and install the packages.
I think it requires some time to achieve that. Please check on Google how to do that.

2) You can connect the Pi to your router with a cable
In case you haven't setup Internet yet. If you are familiar with Debian, you will think that you have to edit the file /etc/network/interfaces, but Raspbmc doesn't use this file!
Instead it is using this file:
/home/pi/.xbmc/userdata/addon_data/script.raspbmc.settings/settings.xml
So edit this file depending of your network. Also edit the value for your Wifi network (this will avoid you to have to modify this file a second time).
Restart your Pi (sudo reboot). Now you should be connected to Internet. It's the first time your Pi has access to Internet, so it will update some files!! So wait a little bit before continuing.

Exit XBMC to go to the terminal and execute the following command to update the firmware:

sudo apt-get install firmware-ralink

Installation is successful, good! Unplug the Ethernet cable and restart your Pi!

Finally the end! Now you should be connected using the WiFi dongle!

Raspberry Pi for beginner - Part 3: Installation

In this post, I'll show you how to prepare the memory card.

Warning: the steps described in this post don't work for the Wifi dongle (Buffalo WLI-UC-GNM2) that I bought. Here is what I had to do.

Which OS to install ?

The RaspberryPi is not limited to one OS. You can install different distributions of Linux.

The most popular one is Raspbian based on debian.

Raspbian is a free operating system based on Debian optimized for the Raspberry Pi hardware. An operating system is the set of basic programs and utilities that make your Raspberry Pi run. However, Raspbian provides more than a pure OS: it comes with over 35,000 packages, pre-compiled software bundled in a nice format for easy installation on your Raspberry Pi. 

Another popular OS is OpenElec.

Open Embedded Linux Entertainment Center (OpenELEC) is a small Linux distribution built from scratch as a platform to turn your computer into an XBMC media center. OpenELEC is designed to make your system boot fast, and the install is so easy that anyone can turn a blank PC into a media machine in less than 15 minutes.

The one I will use is Raspbmc (old name for OSMC)

Raspbmc is a minimal Linux distribution based on Debian that brings XBMC to your Raspberry Pi. This device has an excellent form factor and enough power to handle media playback, making it an ideal component in a low HTPC setup, yet delivering the same XBMC experience that can be enjoyed on much more costly platforms.

Why I choose Raspbmc?

• I like the fact that it is based on Debian, that means a lot of packages are available.
• it's the first one I heard about
• looks like many people are using it, so you can find help on Internet more easily
• based on the list of features (here), I think it can do many things

 It's really easy to install the OS, so in case I don't feel comfortable using Raspbmc, it will be easy to switch to another one.

How do I install Raspbmc ?

Prepare your micro SD card. Make sure there is no data in it, because it will be deleted. Just plug it in your computer.

Download the installer from the official Website. There is an installer for Windows and for Mac/Linux. In this post, I will use the installer for Windows. My machine is Windows 8.

• extract the files (README.txt, setup.exe) somewhere on your Windows
• double click on setup.exe
• UAC (User Account Control) opens, click on YES
• a small popup opens and click on Accept

• on the main window, make sure in the list of device to select the micro SD card !
• check I accept the license agreement
• check Manually configure networking

• a small window will appear, select Wireless
• check Obtain an IP address automatically
• enter your SSID (the name of your Wifi network)
• choose the encryption (for me it's WPA/WPA2)
• enter your wifi password

• click on Save
• back on the main window, click on Install

Wait 1 minute and then you will see this window

Donate money if you think it was easy and you feel happy :)


Next step, I will plug everything and start the Pi! I will explain how to setup the Pi.

Note: XBMC will change name soon! It will be called Kodi! And Raspbmc will also change name for OSMC! So don't get confused!