Posts tagged with: linux

Free extra space for Linux EXT2/3/4 filesystem

Linux ext2/ext3/ext4 (ext from now on) file system will reserved 5% of partition space whenever you create the partition. This reserved space is said to be meant for system admins when space is maxed out and they could release it for emergency usage. The fact this, this space is seldom used and even not known by a lot of users. The 5% space is not much during the time when storage is like 250GB. However, when it comes with modern days with 4TB HDD, this becomes quite a large amount of unused storage waste.

On system partitions, you can leave 1% reserved and it haven’t cause me any problems for the past 5 years. This 5% is a lot when your HDD size is like 3TB. That’s a whopping 150GB of extra space per drive. When you have like 4 drives, it is 600GB!!! And in storage partition, you do not really need the reserved space. Setting it to 0% reserved is almost a no brainer.

So how do you free up the space? Will there be any data loss? Do I have to re-format my partition?

Well, the answer is EASY, NO and NO.

 

For partition already being used:

To free up the space, just run command below:
sudo tune2fs -m <new percentage> /dev/<partitionID>

* new percentage = how many % you want to reserve
** partitionID = how many

[root@web1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md2 267G 37G 217G 15% /
tmpfs 16G 0 16G 0% /dev/shm
/dev/md0 1008M 59M 899M 7% /boot

[root@web1 ~]# tune2fs -m 1 /dev/md2
tune2fs 1.41.12 (17-May-2010)
Setting reserved blocks percentage to 1% (709314 blocks)

[root@web1 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md2 267G 37G 228G 14% /
tmpfs 16G 0 16G 0% /dev/shm
/dev/md0 1008M 59M 899M 7% /boot

Once done, you can continue to use your partition without any issue.

ps. You can see in the example, it works on an mdraid partition as well! 🙂

 

For partition during format:

While using mkfs command to format, you can set the reserved space parameter.

mkfs.ext2 -m 0 /dev/sdc1

Above example set reserved space to 0%.

That’s it for reserved space. Easy and simple work that take less than 5 minutes. Totally useful in NAS environment when you have 4 x 4TB HDD. Whatever the space you save, try to always have redundancy for your storage like RAID1 or RAID5 or RAID6. Speaking of which, I have had a bit of study on this and revamp my storage redundancy strategy. This will be left on an entirely new post which I will do when I have the time.


Setup server to send email directly in Ubuntu without using SMTP gateway

There are a lot of usage for servers such as web hosting, filesharing, application hosting and so on. Most of these servers does no need to have a full fledged email service running in them. However, very often these servers still need to send out emails for example, in the form of alerts such as diskspace full and so on. There are basically 2 ways of doing this,

1) setup server or applications to send email using SMTP login to a proper email server

2) setup server to allow it send out email directly without going through SMTP

The proper way of doing this is using method 1. However it entails a lot of work and I personally prefer method 2. It is much easier to setup and does not rely on SMTP server to send out email. Here, I will show how to setup server to send out email using method 2 on Ubuntu using postfix MTA. It should work for most version of Ubuntu and Debian based system.

apt-get update

apt-get install postfix mailx

During postfix installation, you will be brought to postfix setup screen. If you don’t get that screen, manually type dpkg-reconfigure postfix.

General type of mail configuration: Internet

Just press enter for the rest of the options.

Once this is done, a basic config file will be generated for postfix. You don’t really need to config file. You can just empty it and restart postfix.

cat /dev/null > /etc/postfix/main.cf

/etc/init.d/postfix restart

Test sending out email:

echo testing | mail -s subjecttesting email@youremail.com

Done


Bash scripting: check process running or not and kill it after certain time

I have been doing some bash scripting lately. I am totally noob when it comes to scripting due to the fact that I kinda hate programming. Still, anyone who have done Linux bash scripting would know it is very powerful. Especially if you know awk, it will be even more terror. Anyway, I have this problem that I needed to solve lately. So here goes:

I have a script that run lots rsync commands. I need to be able to kill the rsync command if it take too long to run. Most of the time this is due to the target server have slow connection issue.

To kill the rsync command, I need to know what process ID (PID) it is running as. In bash, you can find out the PID of current running script by adding this command below into the script:

test1.sh
#!/bin/bash
PID=$$
echo $PID

root:# ./test1.sh
21334

There is one problem with this script above (at least for my application). That is actually NOT what I need. Like I said, the script is running rsync command. The command above specifically $$ will only list out the PID of current script. However, I need the PID of the rsync command. Look at example below:

test2.sh 
#!/bin/bash
rsync --dry-run /home /tmp
PID=$$
echo $PID

root:# ./test2.sh
21355

The output process above actually show the PID of test2.sh script. However, I need the PID of the child of this script which is the rsync command. I have searched around and so far the only command that I found to do it is using $!. This $! will actually get the PID of the process that just got send into the background. Therefore we need to send the rsync script into the background.

test3.sh
#!/bin/bash
PID=$$
rsync -a --dry-run /home /tmp &
PID2=$!
echo $PID
echo $PID2

root:# ./test3.sh
21366
21367

There! You can see $PID2 is now showing the PID of the rsync process. If your /home folder have lots of files then the script will take some time to run. Even though you will get your prompt back, the rsync is still running in the background. In that case, you can open another ssh session and use command “ps aux | grep rsync” to see the actual PID which should tally with the value of $PID2.

Now let’s make things more complicated.

Why do I need this $PID2? That is so that I can kill the process if it is taking too long to run. I do not want this one process to be hogging the line while another rsync is waiting next in line to run.

Here is the last part of the script with everything thrown in:

test4.sh
#!/bin/bash
rsync -a --dry-run /home /tmp &
PID2=$!
count=0
waittime=30
while kill -0 $PID2 2> /dev/null
do
sleep 1
((count++))
if [ $count -gt $waittime ] ; then
kill -TERM $PID2 2> /dev/null
break
fi
done
wait ${PID2}
rcode=$?
echo "Return code is $rcode"

Ok, this is the complete code. Run it and see what happens. Play with the settings and see what happens. Again, the rsync will be send to the background however, a counter will be counting and waiting for the script to complete. Hence the do-loop. In the loop, there is a sleep command which actually need to wait 1 second each time. The loop will exit when either the process is complete or the waittime expired. In our example, process will be killed after 30 seconds of waiting. Once the loop is done, notice the command “wait ${PID2}”. This is needed to get the return code for the process ID. In my script, this return code will notify me if the rsync did not complete with success.

I like this script because it is smart and make good use of the limitation of bash scripting. It send the process to the background purely for getting it’s PID using $! command. There are other ways to get this but so far, I like this one as it is simple to understand and deals with less commands like awk and so on.

Reference: http://www.unix.com/shell-programming-scripting/20412-check-if-job-still-alive-killing-after-certain-walltime.html


Linux 64bit or 32bit?

Linux 64bit has been around for a very long time. It came out long before Microsoft released 64bit version of their Windows. During that time, AMD was the leader in 64 bit processor, thus it was called AMD64 during that time. The name has stuck on until now that some linux packages are using it in their package for 64bit packages. So don’t be puzzled that you will be installing package_AMD64.deb for your debian or ubuntu.

It was a no brainier for me by now that Linux 64bit platform has been established to be very stable and almost all packages comes with it. Being 64bit means your software will have double the bandwidth for memory usage and even processing power. However, in real work, performance is not that much faster. But the main thing about 64bit platform is to break the memory limitation of 32bit which is 4GB RAM.

I would go for 64bit for almost all my OS installation nowadays be it Linux or Windows. However, something happened the other day that made me re-think my perception on 64bit.

Recently, there was a server that I manage that needed to be upgraded. We bought a new server with better hardware specs but it remained 4GB RAM. At that time, we do not think there is a need to add more RAM and server RAMs are not cheap either. As usual, I went for Ubuntu 10.04 LTS 64bit as this is currently our choice of Linux for our servers. After taking few days to painstakingly install, configure and migrated the data from old server, the new server went online. However, the following day we encountered some problems. The server were slow and at one time it even crashed. We had to reboot it and as usual, we check the logs for any signs of problems. Eventually, we found that the APC (Alternative PHP cache) that is running in the server keep churning out error that it does not have enough memory. So we knew this has something to do with memory. We tinkered with APC and the server became more stable and load is now running pretty low at less than 2.0.

However, there are still some problem which we are still not sure whether it is APC or other components like Apache2 or PHP. Eventhough the configurations are all same from old server, the software version is running at newer version. Another few days are spend checking and tinkering until we found the “blank page” problem is due to APC. However, disabling APC does not solve the problem permanently as it will cause the server to run higher load and more memory.

After even more checking, we found that each PHP session is using double the amount of memory from previous server. This eventually lead to the culprit…..64bit. Apparently when you go to 64bit, the amount of memory used will almost become double. That is for certain applications only. In our case, PHP is using double the memory to run the usual stuff but to give 64bit some credit, it does run a bit faster than before. Maybe it is due to the wider 64bit memory bandwidth or maybe it is due to the better processor.

In any case, this taught me a lesson on 32bit vs 64bit.

  1. Use 32bit – if you do not plan to upgrade your system RAM to more than 4GB.**
  2. Use 64bit – if you will definitely be using more than 4GB memory now or later in upgrade. To take advantage of 64bit processor and hardware.

**Bear in mind that it is possible to have 32bit Linux with more than 4GB RAM. Linux 32bit kernel will still be able to make use of all the memory using something called PAE (Physical Address Extension). However, this method does have some limitations. **