SmartOS first steps

Recently I was looking at SmartOS because I read an article on the net.
At first, I didn't quite understand what it was exactly. Then I decided to install it and see what it is.

So here is some information from a beginner point of view. What I will say is maybe not 100% accurate, but I will try my best. The goal of this post is to show you how to start playing with SmartOS. What I will show is just good for testing purposes!

Here is a quick explanation for someone who is used to work with VirtualBox:

SmartOS is like if VirtualBox is transformed into a OS.

And from Wikipedia:

SmartOS is a free and open-source SVR4 hypervisor, based on the UNIX operating system that combines OpenSolaris technology with Linux's KVM virtualization.

With SmartOS you can create virtual machines really easily. It is an operating system with the very minimum. The size of the ISO is around 155Mb.

In the setup described below, we will use VirtualBox to host a virtual machine using SmartOS called SmartOS Manager.

Here is a little schema:

So "SmartOS manager" is a guest (for VirtualBox) and it is also a host to manage SmartOS VMs.

How to setup something really basic

• install VirtualBox
• download the latest ISO file for SmartOS
• open VirtualBox and create a new VM with these settings
• Name: SmartOS manager
• Type: Solaris 
• Version: Oracle Solaris 11 (64-bit)
• 2GB of ram
• VDI disk of 16GB

Don't start the VM yet. Go to the settings of this VM and click on Network. Then select:

• Attached to: Bridged Adapter
• Adapter type: Intel PRO/1000 MT Desktop
• Promiscuous Mode: Allow All

Now you can start the VM. It will ask you to select the installation disk. Just choose the ISO file downloaded earlier.

SmartOS will ask you some questions about your setup:

• IP address
• Default gateway
• DNS
• NTP server
• Host
• Default DNS search Domain
• Password for root account
• Data pool (where VM's will be installed), just type "y"

That's it! Now you should see that:

The system will now finish configuration and reboot. Please wait...
rebooting...

Great. Now let's see how to create a VM. By the way, in the world of SmartOS, a VM is called a zone.

Now, ssh to the VM and run the command to see all pre-configured zones:

$ imgadm avail
UUID                                  NAME     VERSION     OS       TYPE          PUB
f669428c-a939-11e2-a485-b790efc0f0c1  base     13.1.0      smartos  zone-dataset  2013-04-26
9eac5c0c-a941-11e2-a7dc-57a6b041988f  base64   13.1.0      smartos  zone-dataset  2013-04-26
...

As you can see there are a lot available. The one that we will use is (because it is the latest "base-64"):

96bcddda-beb7-11e5-af20-a3fb54c8ae29  base-64-lts  15.4.0  smartos  zone-dataset  2016-01-19

Import (download) the VM and wait a little bit:

$ imgadm import 96bcddda-beb7-11e5-af20-a3fb54c8ae29

Create a VM manifest file:

$ vi /root/web01.json
{
 "brand"              : "joyent",
 "image_uuid"         : "5c7d0d24-3475-11e5-8e67-27953a8b237e",
 "alias"              : "web01",
 "hostname"           : "web01",
 "max_physical_memory": 512,
 "quota"              : 7,
 "resolvers"          : ["8.8.8.8", "208.67.220.220"],
 "nics"               : [
                         {
                           "nic_tag": "admin",
                           "ip"     : "10.88.88.52",
                           "netmask": "255.255.255.0",
                           "gateway": "10.88.88.2"
                         }
                        ]
}

Resolvers: set the same DNS servers as you did for the SmartOS manager

IP: use an IP in the same network as the SmartOS manager

Gateway: same as the SmartOS manager

Create the VM:

$ vmadm create -f web01.json

Now you can list the existing VM;

$ vmadm list
UUID                                  TYPE  RAM      STATE             ALIAS
25ac9366-21ad-4942-a98d-f35373c3cf40  OS    512      running           web01

And to login:

$ zlogin 25ac9366-21ad-4942-a98d-f35373c3cf40

From there, you should be able to ping google:

$ ping google.com
google.com is alive

Now you can play with this VM. To install packages:

$ pkgin up
$ pkgin in vim
$ pkgin in gcc47
$ pkgin in gmake

As a sidenote, to install Java, you can download the archive from Oracle website. In my case, I used the Solaris x86 64-bit JDK.

To exit zlogin, press "~." (without double quotes).

And that's it! You have setup a basic SmartOS VM! Nothing fancy but I hope you have a better idea on how to start using SmartOS.

For more information:
https://docs.joyent.com/public-cloud/instances/infrastructure/images/smartos/managing-a-smartos-instance/installing-software
https://wiki.smartos.org/display/DOC/How+to+create+a+zone+%28+OS+virtualized+machine+%29+in+SmartOS
http://troysunix.blogspot.jp/2013/02/intro-smartos-setup-pt-3.html

How to delete a VirtualBox VM using command line

VirtualBox comes with a GUI that let you easily manage your virtual machines.
However, you may have no Desktop and have to manage your virtual machines via command line.

There is a great tools called vboxmanage that let you do anything you want. I will show you how to completely delete a virtual machine with vboxmanage.

First you should get the name of the VM you want to delete:

$ vboxmanage list vms
"MyVM" {bc0f9b2a-1c49-4771-8179-610a5bf10578}
"My second VM" {a2b9f0ea-3c48-4770-8179-610a5bf10578}
"My third VM" {9b2aae0f-6c48-4771-8179-610a5bf10578}

Identify the VM you want to delete and make sure you don't delete the wrong one, because once you execute the following command, there is no confirmation.

Here is what will do the following command:

•     all hard disk image files, including differencing files, which are used by the machine and not shared with other machines;
•     saved state files that the machine created, if any (one if the machine was in "saved" state and one for each online snapshot);
•     the machine XML file and its backups;
•     the machine log files, if any;
•     the machine directory, if it is empty after having deleted all the above.

Ready to run the command to delete the VM? Here is the command:

$ vboxmanage unregistervm < name or uid > --delete

For example:

$ vboxmanage unregistervm MyVM --delete
$ vboxmanage unregistervm "My second VM" --delete
$ vboxmanage unregistervm 9b2aae0f-6c48-4771-8179-610a5bf10578 --delete

Remember there is no confirmation, as soon as you press enter the VM will be deleted.

Source: https://www.virtualbox.org/manual/ch08.html#vboxmanage-registervm

How to replace a disk in RAID6

On one of the server, a disk had some problems so it was time to change it.
Machine is Ubuntu 10.04 with 4 disks in RAID 6 and using LVM.

First, let's "remove" the failing disk (sdb2) from the RAID:

$ mdadm --manage /dev/md0 --fail /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md0

$ mdadm --manage /dev/md0 --remove /dev/sdb2
mdadm: hot removed /dev/sdb2 from /dev/md0

Now if you run lsblk you can see that the disk doesn't belong to the RAID.

-> shutdown the machine
-> replace the disk
-> start the machine

When the machine started, there was a problem with grub and grub rescue command prompt showed up.

To avoid to mess with grub, I decided to use Ubuntu Desktop live CD. I download it on a USB stick and boot the machine on it.

I had the following issue:

missing parameter in configuration file. keyword path

To fix it, simply type live and press enter.
Note: if you hit tab, you can see a list of options.

Perfect, it is working. Now let's install some packages:

$ sudo apt-get install mdadm lvm2

Let's create the partition of the new disk (didn't work for me):

$ parted -a optimal /dev/sdb
(parted) mklabel gpt
(parted) mkpart primary 1 2
(parted) set 1 bios_grub on
(parted) mkpart primary 2 -1
(parted) set 2 raid on
(parted) print

With this method, there was a small unallocated partition at the end... So I couldn't add the disk to the RAID because the size of sdb2 was too small.

Instead I used dd.
First you need to calculate the count parameter.
count = (128*N)+1024
Where N is the number of partitions you have. In this case I had 2, so the result is 1280.

The following commands will copy the partition table from /dev/sda to /dev/sdb. Make sure you type the second command correctly!

$ dd if=/dev/sda of=GPT_TABLE bs=1 count=1280
$ dd if=GPT_TABLE of=/dev/sdb bs=1 count=1280

Then I had to reboot the machine.

Finally add /dev/sdb2 to the RAID:

$ mdadm --manage /dev/md0 --add /dev/sdb2
mdadm: added /dev/sdb2

Then it will take time to repair everything. You can see the process using this command:

$ cat /proc/mdstat

My original disks are Seagate and I tried to add a Western Digital. Unfortunately the size was different by 1M, so I couldn't use the WD disk. I ordered a similar Seagate disk, and the size was OK.
The disk couldn't be added because the size was smaller... and by not much. For example(fdisk -l):

Seagate Barracuda /dev/sda 2,000,396,746,752 bytes
Western Digital   /dev/sdb 2,000,395,698,176 bytes

The best solution is : never us the full disk! Leave a 10 or 15M at the end. So disks with small differences in size can be added.

Add node to Puppet

In this post, I will explain how to add a node (computer) to Puppet.

I assume you have a Puppet master server running and it can ping the future node agent.

Puppet master server:

• Debian 6.0
• Name: master-puppet
• Puppet version: 3.7.4

Future puppet node agent:

• Ubuntu server 14.04 TLS
• Name: new-node

NOTE: Puppet version 4.0 will be out soon, and the process may be different.

On the AGENT

Download puppet:

wget --no-check-certificate https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

Note: if you use a different OS, please download the appropriate file

Install Puppet:

dpkg -i puppetlabs-release-trusty.deb
apt-get update
apt-get install puppet

Make Puppet to start on boot. Edit the file /etc/default/puppet:
START=yes

Run Puppet:

puppet agent --no-daemonize --onetime --test
Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
Info: Creating a new SSL key for new-node
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for new-node
Info: Certificate Request fingerprint (SHA256): 4A:...:04
Info: Caching certificate for ca
Exiting; no certificate found and waitforcert is disabled

On the MASTER

Sign the certificate created by the node agent:

puppet cert sign new-node
Notice: Signed certificate request for new-node
Notice: Removing file Puppet::SSL::CertificateRequest new-node at '/var/lib/puppet/ssl/ca/requests/new-node.pem'

Add the node info in the file site.pp of Puppet.

on the NODE

Run puppet:

puppet agent --no-daemonize --onetime --test

And that's it!

reducing the advertised EDNS UDP packet

I had to set up a PPPoE connection on my Ubuntu server. With the help of the command pppoeconf, it is kind of straigtforward.

However, when browsing some Websites I could see in the log file:

... after reducing the advertised EDNS UDP packet size to 512 octets

And even worse, some Websites were not showing up (ex: finance.yahoo.co.jp).

Solution

There were 2 problems:

1) Problem with IPv6
I had to disable IPv6 (probably a problem with my ISP?) with lines in /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1 

Then run:

sysctl -p /etc/sysctl.conf

2) Problem with packet size
A rule for iptables was automatically created in /etc/ppp/if-up.d/0clampmss. And later on I was flushing all the rules to set my own rules (set in another file). Because I was not re-running 0clampsmss then the rule was not added.
And this rule was for resizing the packet.
So in my own rules, I stopped flushing the table mangle.

And that's it.

How to mount a single disk that was part of RAID 1

I screwed up and deleted the folder /etc of our main computer (samba server, dhcp, router, etc.). Terrible mistake...

The positive thing is that it was an old Ubuntu 8.04 and we wanted to update it.
So here was the time to do it.

On the machine, there was 2 hard drives in RAID 1 (mirror). So we removed them and put 2 new drives. We installed Ubuntu 14.04 and all the necessary packages.

Then it was time to copy data from the old disks to the new ones.
I didn't want to lose any more data, so I wanted to connect only 1 hard drive.

The problem is that this hard drive was part of a RAID, it is not a normal hard drive that we can mount as usual.

If you try, you will get error like:

mount: unknown filesystem type 'linux_raid_member'
mount: /dev/sdc2 already mounted or /mnt/recovery busy

Solution

Here are the steps to do:

1)  create a mount point

mkdir /mnt/recovery

2) list the disk information

fdisk -l

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000abbde

Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *          32    62500863    31250416   fd  Linux raid autodetect
/dev/sdc2      1937904885  1953520064     7807590   fd  Linux raid autodetect
/dev/sdc3        62500864  1937899519   937699328   fd  Linux raid autodetect

I want to mount /dev/sdc3. The important information that we need in the next step:

• sectors: 512
• start: 62500864

3) finally run these commands:

$ losetup --find --show --read-only --offset $((62500864*512)) /dev/sdc
/dev/loop3

$ fsck.ext3 -n -v /dev/loop3
e2fsck 1.42.9 (4-Feb-2014)
Warning: skipping journal recovery because doing a read-only filesystem check.

$ file -s /dev/loop3
/dev/loop2: Linux rev 1.0 ext3 filesystem data, UUID=dbf4-...-9a98a (needs journal recovery) (large files)

$ mount -t ext3 -o ro,noload /dev/loop3 /mnt/recovery

note: I'm using loop3, but it can be a different number in your case.

And that's it! you can now access your files.


If you have an error like:

mount: wrong fs type, bad option, bad superblock on ....

It probably means you have entered a wrong offset.

In case of error, use the following command to see the log:

dmesg | tail

Another solution on a computer without any RAID

On my desktop, I have no RAID disk, so I could run the following command:

mdadm --examine --scan

And that's it!


Here are 2 links that helped me a lot to find the solution!
http://digital-forensics.sans.org/blog/2011/06/14/digital-forensics-mounting-dirty-ext4-filesystems
http://unix.stackexchange.com/questions/72279/how-do-i-recover-files-from-a-single-degraded-mdadm-raid1-drive-not-enough-to

How to send an email after a push to gitolite

This is for the gitolite package on Debian.

# cat /etc/gitolite/VERSION
1.5.4-2+squeeze1 (Debian)

I assume gitolite is working correctly. You can commit, push and pull.

Log in as the gitolite user:

su - gitolite

In my case $HOME is /var/lib/gitolite

Go to the folder $HOME/.gitolite/hooks/common

If the file post-receive exists, then create a copy somewhere else.

cp post-receive /tmp/post-receive

Copy the hook to send an email and rename it:

cp /usr/share/gitolite/hooks/common/post-receive-email post-receive

Make sure privileges are OK:

chowm gitolite:gitolite post-receive
chmod 755 post-receive

Edit $HOME/.gitolite.rc

# edit this line:
$GL_GITCONFIG_KEYS = "";
# for this:
$GL_GITCONFIG_KEYS = ".*";

Then run this to apply the hook to all the repositories:

gl-setup

Now look at one of your repositories. In my case:

ls $HOME/repositories/my_repo.git/hooks

Then you should see the file post-receive.

Finally edit $HOME/.gitolite/conf/gitolite.conf

repo my_repo
    RW+  = user1
    config hooks.mailinglist = destination@test.com
    config hooks.emailprefix = "[my_repo] "
    config hooks.envelopesender = from@test.com
 

Now you can push something to my_repo and should receive an email. Make sure you have setup
an email server correctly.

if you see this message when you push something:

remote: sed: can't read ./description: No such file or directory

It is just a warning. You have successfully pushed your code, but in the email you received, you will see:

It was generated because a ref change was pushed to the repository containing
the project "".

See the empty double quote at the end?

If you want to fix this and the warning message, simply create a file:

vi $HOME/respositorirs/my_repo/description

And inside the file write something, for example:

my_repo

Then if everything is good, the email will look like:

It was generated because a ref change was pushed to the repository containing
the project "my_repo".