Tuesday, March 3, 2015

How to replace a disk in RAID6

On one of the server, a disk had some problems so it was time to change it.
Machine is Ubuntu 10.04 with 4 disks in RAID 6 and using LVM.

First, let's "remove" the failing disk (sdb2) from the RAID:
$ mdadm --manage /dev/md0 --fail /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md0

$ mdadm --manage /dev/md0 --remove /dev/sdb2
mdadm: hot removed /dev/sdb2 from /dev/md0

Now if you run lsblk you can see that the disk doesn't belong to the RAID.

-> shutdown the machine
-> replace the disk
-> start the machine

When the machine started, there was a problem with grub and grub rescue command prompt showed up.

To avoid to mess with grub, I decided to use Ubuntu Desktop live CD. I download it on a USB stick and boot the machine on it.

I had the following issue:
missing parameter in configuration file. keyword path
To fix it, simply type live and press enter.
Note: if you hit tab, you can see a list of options.

Perfect, it is working. Now let's install some packages:
$ sudo apt-get install mdadm lvm2

Let's create the partition of the new disk (didn't work for me):
$ parted -a optimal /dev/sdb
(parted) mklabel gpt
(parted) mkpart primary 1 2
(parted) set 1 bios_grub on
(parted) mkpart primary 2 -1
(parted) set 2 raid on
(parted) print

With this method, there was a small unallocated partition at the end... So I couldn't add the disk to the RAID because the size of sdb2 was too small.

Instead I used dd.
First you need to calculate the count parameter.
count = (128*N)+1024
Where N is the number of partitions you have. In this case I had 2, so the result is 1280.

The following commands will copy the partition table from /dev/sda to /dev/sdb. Make sure you type the second command correctly!

$ dd if=/dev/sda of=GPT_TABLE bs=1 count=1280
$ dd if=GPT_TABLE of=/dev/sdb bs=1 count=1280

Then I had to reboot the machine.

Finally add /dev/sdb2 to the RAID:
$ mdadm --manage /dev/md0 --add /dev/sdb2
mdadm: added /dev/sdb2

Then it will take time to repair everything. You can see the process using this command:
$ cat /proc/mdstat

My original disks are Seagate and I tried to add a Western Digital. Unfortunately the size was different by 1M, so I couldn't use the WD disk. I ordered a similar Seagate disk, and the size was OK.
The disk couldn't be added because the size was smaller... and by not much. For example(fdisk -l):
Seagate Barracuda /dev/sda 2,000,396,746,752 bytes
Western Digital   /dev/sdb 2,000,395,698,176 bytes

The best solution is : never us the full disk! Leave a 10 or 15M at the end. So disks with small differences in size can be added.

Monday, February 23, 2015

Add node to Puppet

In this post, I will explain how to add a node (computer) to Puppet.

I assume you have a Puppet master server running and it can ping the future node agent.

Puppet master server:
  • Debian 6.0
  • Name: master-puppet
  • Puppet version: 3.7.4

Future puppet node agent:
  • Ubuntu server 14.04 TLS
  • Name: new-node

NOTE: Puppet version 4.0 will be out soon, and the process may be different.

On the AGENT

Download puppet:
wget --no-check-certificate https://apt.puppetlabs.com/puppetlabs-release-trusty.deb
Note: if you use a different OS, please download the appropriate file

Install Puppet:
dpkg -i puppetlabs-release-trusty.deb
apt-get update
apt-get install puppet

Make Puppet to start on boot. Edit the file /etc/default/puppet:

Run Puppet:
puppet agent --no-daemonize --onetime --test
Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
Info: Creating a new SSL key for new-node
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for new-node
Info: Certificate Request fingerprint (SHA256): 4A:...:04
Info: Caching certificate for ca
Exiting; no certificate found and waitforcert is disabled


Sign the certificate created by the node agent:
puppet cert sign new-node
Notice: Signed certificate request for new-node
Notice: Removing file Puppet::SSL::CertificateRequest new-node at '/var/lib/puppet/ssl/ca/requests/new-node.pem'

Add the node info in the file site.pp of Puppet.

on the NODE

Run puppet:
puppet agent --no-daemonize --onetime --test

And that's it!

reducing the advertised EDNS UDP packet

I had to set up a PPPoE connection on my Ubuntu server. With the help of the command pppoeconf, it is kind of straigtforward.

However, when browsing some Websites I could see in the log file:
... after reducing the advertised EDNS UDP packet size to 512 octets
And even worse, some Websites were not showing up (ex: finance.yahoo.co.jp).


There were 2 problems:

1) Problem with IPv6
I had to disable IPv6 (probably a problem with my ISP?) with lines in /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1 

Then run:
sysctl -p /etc/sysctl.conf

2) Problem with packet size
A rule for iptables was automatically created in /etc/ppp/if-up.d/0clampmss. And later on I was flushing all the rules to set my own rules (set in another file). Because I was not re-running 0clampsmss then the rule was not added.
And this rule was for resizing the packet.
So in my own rules, I stopped flushing the table mangle.

And that's it.

Wednesday, February 18, 2015

How to mount a single disk that was part of RAID 1

I screwed up and deleted the folder /etc of our main computer (samba server, dhcp, router, etc.). Terrible mistake...

The positive thing is that it was an old Ubuntu 8.04 and we wanted to update it.
So here was the time to do it.

On the machine, there was 2 hard drives in RAID 1 (mirror). So we removed them and put 2 new drives. We installed Ubuntu 14.04 and all the necessary packages.

Then it was time to copy data from the old disks to the new ones.
I didn't want to lose any more data, so I wanted to connect only 1 hard drive.

The problem is that this hard drive was part of a RAID, it is not a normal hard drive that we can mount as usual.

If you try, you will get error like:

mount: unknown filesystem type 'linux_raid_member'
mount: /dev/sdc2 already mounted or /mnt/recovery busy


Here are the steps to do:

1)  create a mount point
mkdir /mnt/recovery

2) list the disk information
fdisk -l

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000abbde

Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *          32    62500863    31250416   fd  Linux raid autodetect
/dev/sdc2      1937904885  1953520064     7807590   fd  Linux raid autodetect
/dev/sdc3        62500864  1937899519   937699328   fd  Linux raid autodetect

I want to mount /dev/sdc3. The important information that we need in the next step:
  • sectors: 512
  • start: 62500864

3) finally run these commands:
$ losetup --find --show --read-only --offset $((62500864*512)) /dev/sdc

$ fsck.ext3 -n -v /dev/loop3
e2fsck 1.42.9 (4-Feb-2014)
Warning: skipping journal recovery because doing a read-only filesystem check.

$ file -s /dev/loop3
/dev/loop2: Linux rev 1.0 ext3 filesystem data, UUID=dbf4-...-9a98a (needs journal recovery) (large files)

$ mount -t ext3 -o ro,noload /dev/loop3 /mnt/recovery
note: I'm using loop3, but it can be a different number in your case.

And that's it! you can now access your files.

If you have an error like:
mount: wrong fs type, bad option, bad superblock on ....
It probably means you have entered a wrong offset.

In case of error, use the following command to see the log:
dmesg | tail

Another solution on a computer without any RAID

On my desktop, I have no RAID disk, so I could run the following command:

mdadm --examine --scan

And that's it!

Here are 2 links that helped me a lot to find the solution!

Wednesday, February 11, 2015

How to send an email after a push to gitolite

This is for the gitolite package on Debian.

# cat /etc/gitolite/VERSION
1.5.4-2+squeeze1 (Debian)

I assume gitolite is working correctly. You can commit, push and pull.

Log in as the gitolite user:
su - gitolite

In my case $HOME is /var/lib/gitolite

Go to the folder $HOME/.gitolite/hooks/common

If the file post-receive exists, then create a copy somewhere else.
cp post-receive /tmp/post-receive

Copy the hook to send an email and rename it:
cp /usr/share/gitolite/hooks/common/post-receive-email post-receive

Make sure privileges are OK:
chowm gitolite:gitolite post-receive
chmod 755 post-receive

Edit $HOME/.gitolite.rc

# edit this line:
# for this:

Then run this to apply the hook to all the repositories:

Now look at one of your repositories. In my case:
ls $HOME/repositories/my_repo.git/hooks

Then you should see the file post-receive.

Finally edit $HOME/.gitolite/conf/gitolite.conf
repo my_repo
    RW+  = user1
    config hooks.mailinglist = destination@test.com
    config hooks.emailprefix = "[my_repo] "
    config hooks.envelopesender = from@test.com
Now you can push something to my_repo and should receive an email. Make sure you have setup
an email server correctly.

if you see this message when you push something:
remote: sed: can't read ./description: No such file or directory

It is just a warning. You have successfully pushed your code, but in the email you received, you will see:
It was generated because a ref change was pushed to the repository containing
the project "".
See the empty double quote at the end?
If you want to fix this and the warning message, simply create a file:
vi $HOME/respositorirs/my_repo/description

And inside the file write something, for example:
Then if everything is good, the email will look like:
It was generated because a ref change was pushed to the repository containing
the project "my_repo".