Category Archives: tech

Technical and how-to articles

Unix, Linux and Mac OS X Notes

Posted on by

Here’s some notable command syntax I use. You can also select the Notes category and you’ll get more specific topics such as Linux LVM and Mac OS X commands.

rsyslog options

Forward syslog events to external host via UDP:
– edit /etc/rsyslog.conf .. add a stanza like the example at the end of the file .. a single @ = UDP forward, @@ = TCP forward

$WorkDirectory /var/lib/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* @10.0.0.45:514

– restart the rsyslog daemon
systemctl restart rsyslog.service
or
service rsyslog restart

Mac OS X syslog to remote syslog server

Forward syslog events on Mac OS X 10.11 to external syslog server via UDP or TCP:
– edit /etc/syslog.conf .. add a line at the end of the file .. a single @ = UDP forward, @@ = TCP forward

*.* @10.0.0.45:514
# remote host is: name or ip:port, e.g. 10.0.0.45:514, port optional

– restart the OS X syslog daemon
sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist
sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist

Write ISO image to USB on Mac

– plug in USB to Mac
– lookup disk number
sudo diskutil list
– unmount the USB
sudo diskutil unmountDisk /dev/disk2
– copy ISO image to USB
sudo dd if=CentOS.iso of=/dev/disk2

NIC MAC change

Changing MAC address of NIC
– RedHat stores this in: /etc/sysconfig
networking/devices/ifcfg-eth?
networking/profiles/default/ifcfg-eth?
hwconf
You need to edit the hwaddr in /etc/sysconfig/hwconf and HWADDR in the other locations (some are links).

ssh tunneling of syslog traffic

– Example SSH configuration for tunneling a syslog TCP stream from a remote server back to a local node:

Remote node has TCP client process (rsyslog) running, we want it to write to a local TCP port (15514/tcp), and have that local port forward to the local node we have initiated the ssh connection from to a syslog daemon listening on port 1514/tcp:

Remote node rsyslog.conf:
@@localhost:15514

Event flow is through ssh on the remote node, listening on 15514/tcp and forwarding to the local node via ssh tunnel launched on the local node:
$ ssh -R 15514:localhost:1514 remotehostusername@remote.hostname.domain

To complete the picture, we probably want some sort of process on the local node to detect when the ssh connection has been lost and (1) re-establish the ssh connection, (2) restart rsyslog on the remote host to re-establish the connection from the remote rsyslog daemon to the ssh listener on port 15514/tcp.

YUM Software Repository

– Manually add DVD location/repository by:

35.3.1.2. Using a Red Hat Enterprise Linux Installation DVD as a Software Repository

To use a Red Hat Enterprise Linux installation DVD as a software repository, either in the form of a physical disc, or in the form of an ISO image file.

1. Create a mount point for the repository:
mkdir -p /path/to/repo

Where /path/to/repo is a location for the repository, for example, /mnt/repo. Mount the DVD on the mount point that you just created. If you are using a physical disc, you need to know the device name of your DVD drive. You can find the names of any CD or DVD drives on your system with the command cat /proc/sys/dev/cdrom/info. The first CD or DVD drive on the system is typically named sr0. When you know the device name, mount the DVD:
mount -r -t iso9660 /dev/device_name /path/to/repo
For example: mount -r -t iso9660 /dev/sr0 /mnt/repo

If you are using an ISO image file of a disc, mount the image file like this:
mount -r -t iso9660 -o loop /path/to/image/file.iso /path/to/repo
For example: mount -r -o loop /home/root/Downloads/RHEL6-Server-i386-DVD.iso /mnt/repo

Note that you can only mount an image file if the storage device that holds the image file is itself mounted. For example, if the image file is stored on a hard drive that is not mounted automatically when the system boots, you must mount the hard drive before you mount an image file stored on that hard drive. Consider a hard drive named /dev/sdb that is not automatically mounted at boot time and which has an image file stored in a directory named Downloads on its first partition:

mkdir /mnt/temp
mount /dev/sdb1 /mnt/temp
mkdir /mnt/repo
mount -r -t iso9660 -o loop mount -r -o loop /mnt/temp/Downloads/RHEL6-Server-i386-DVD.iso /mnt/repo

2. Create a new repo file in the /etc/yum.repos.d/ directory:
The name of the file is not important, as long as it ends in .repo. For example, dvd.repo is an obvious choice. Choose a name for the repo file and open it as a new file with the vi text editor. For example:

vi /etc/yum.repos.d/dvd.repo

[dvd]
baseurl=file:///mnt/repo/Server
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

The name of the repository is specified in square brackets — in this example, [dvd]. The name is not important, but you should choose something that is meaningful and recognizable. The line that specifies the baseurl should contain the path to the mount point that you created previously, suffixed with /Server for a Red Hat Enterprise Linux server installation DVD, or with /Client for a Red Hat Enterprise Linux client installation DVD. NOTE: After installing or upgrading software from the DVD, delete the repo file that you created to get updates from the online sources.

IP Networking

– Manually add IPv4 alias to interface by:
ip addr add 192.168.0.30/24 dev eth4
– Manually remove that IPv4 alias to interface by (note the subnet mask):
ip addr del 192.168.0.30/32 dev eth4
– Manually add route for specific host:
route add -host 45.56.119.201 gw 10.20.1.5

pcap files

– Split large pcap file by using command line tool that comes with Wireshark editcap:
editcap -c 10000 infile.pcap outfile.pcap

tcpdump options

Display only packets with SYN flag set (for host 10.10.1.1 and NOT port 80):
tcpdump 'host 10.10.1.1  &&  tcp[13]&0x02 = 2  &&  !port 80'

Mac OS X (10.7)

sudo /usr/sbin/sysctl -w net.inet.ip.fw.enable=1
sudo /sbin/ipfw -q /etc/firewall.conf
sudo ifconfig en0 lladdr 00:1e:c2:0f:86:10
sudo ifconfig en1 alias 192.168.0.10 netmask 255.255.255.0
sudo ifconfig en1 -alias 192.168.0.10
sudo route add -net 10.2.1.0/24 10.3.1.1

rpm commands:

List files in an rpm file
rpm -qlp package-name.rpm

List files associated with an already installed package
rpm --query –-filesbypkg package-name
How do I find out what rpm provides a file?
yum whatprovides '*bin/grep'
Returns the package that supplies the file, but the repoquery tool (in the yum-utils package) is faster and provides more output as well as do other queries such as listing package contents, dependencies, reverse-dependencies.

sed commands:

Remove specific patterns (delete or remove blank lines):
sed '/^$/d'
sed command matching multiple line pattern (a single log line got split into two lines, the second line beginning with a space):
cat syslog3.txt | sed 'N;s/\n / /' > syslog3a.txt
– matches the end of line (\n) and space at the beginning of the next line, then removes the newline

awk commands:

Print out key value pairs KVP separated by =:
awk /SRC=/ RS=" "
Print out source IP for all iptables entries that contain the keyword recent:
cat /var/log/iptables.log | egrep recent | awk /SRC=/ RS=" " | sort | uniq
Sum column one in a file, giving the average (where NR is the automatically computed number of lines in the file):
./packet_parser analyzer_data.pcap | awk '{print $5}' | sed -e 's/length=//g' | awk 'BEGIN {sum=0} { sum+=$1 } END { print sum/NR }'
Find the number of tabs per line – used to do a sanity check on tab delimited input files
awk -F$'\t' '{print NF-1;}' file | sort -u

sort by some mid-line column

I wanted to sort by the sub-facility message name internal to the dovecot messages, so found the default behavior of sorting by space delimited columns works.

sort -k6 refers to the sixth column with the default delimiter as space.
sort -tx -k1.20,1.25 is an alternative, where ‘x’ is a delimiter character that does not appear anywhere in the line, and character position 20 is the start of the sort key and character position 25 is the end of the sort key.

This sorts by the bold column:
$ sort -k6 dovecot.txt
Oct 7 09:09:31 server1 dovecot: auth: mysql: Connected to 10.30.132.15 (db1)
Oct 7 09:34:03 server1 dovecot: auth: sql(user1@example.com,10.30.132.15): Password mismatch
Oct 7 09:33:36 server1 dovecot: auth: sql(someuser@example.com,10.30.132.15): unknown user
Oct 7 09:15:27 server1 dovecot: imap(user1@example.com): Disconnected for inactivity bytes=946/215256
Oct 7 09:21:11 server1 dovecot: imap(user2@example2.com): Disconnected: Logged out bytes=120/12718

dos2unix equivalent with tr

tr -d '\15\32' < windows-file.csv > unix-file.csv

Fedora 16 biosdevname

– Fedora 16 includes a package called “biosdevname” that sets up strange network port names (p3p1 versus eth0) .. since I don’t particilarly care if my ethernet adapter(s) is(are) in a particular PCI slot, remove this nonsense by:

yum erase biosdevname

– to take total control of network interfaces back over (edit /etc/sysconfig/network-scripts/ifcfg-eth?)

– remove NetworkManager

yum erase NetworkManager
chkconfig network on

WordPress notes for pomeroy.us

Posted on by

Production site is www.pomeroy.us
Development site is dev.pomeroy.us

Assumptions:
– webserver root directory is /var/web
– production node is called prod
– development node is called dev
– WordPress database is called wpdb

Procedure to copy production WordPress instance to the development node:
1. Copy webserver www root dir via a tarball
tar czf prod-20110808.tgz /var/web

2. Dump the WordPress database to a MySQL dmp file:
mysqldump -u$mysqluser -p$mysqlpass wpdb | \
 gzip -c > prod-20110808.dmp.gz

3. Copy these two backup files to the dev node:
scp prod-20110808* user@dev:.

On the development node:
4. Unpack the webserver tarball:
mv /var/web /var/web.previous
cd /
tar xzvf prod-20110808.tgz

5. Drop the WordPress database and restore the new version:
mysql> drop database wpdb;
mysql> create database wpdp;
$ gunzip prod-20110808.dmp.gz
$ mysql -u$mysqluser -p wpdb < prod-20110808.dmp

6. Update the WordPress 'siteurl' and 'home' options to point to the development node:
update wp_options set option_value='http://dev.pomeroy.us' where option_name='siteurl';
update wp_options set option_value='http://dev.pomeroy.us' where option_name='home';

Should be all done!

Building a new PVR

Posted on by

<Updated Aug 18, 2011 after a successful PVR rollout>

Technology has evolved since the last MythTV PVR I built, as chronicled here.  Here’s the latest techniques and tech that I’ve used to (start) build(ing) my current PVR. I’ll update this article as I go, as there’s been some bumps along the way, so completion of the project has been slower than I anticipated.

Requirements for my new PVR include:

  • Linux operating system for cost and flexibility reasons
  • Quiet! Fan-less operation if at all possible, external power supply ok
  • Small form factor, black case to fit in with my current home theater gear
  • Video capture with MPEG-2 hardware acceleration to help keep the CPU needed as small as possible, in an expansion card format for the most compact physical footprint .. additionally there must be at least two independent tuners
  • Analog tuners, but would be good if they were capable of digital for when I eventually move to digital/HD
  • IR receiver and transmitter capability for easy remote control and ability of the PVR to use my current set-top box as a source (gives me all the cable company movies and channels that are not available via the basic cable connection
  • Ability to schedule at least 10 shows and retain 5 episodes of each show .. also ability to schedule based on show name alone
  • Ability to perform post-recording processing, such as removing commercials or changing formats
  • Should be able to use a pre-packaged distribution for most if not all of the functions .. I know it’s a home-brew, but I’m tired of messing with individual packages, firmware, and custom codes to make it work. Using a distribution package makes it easier to maintain through updates.
  • Want to purchase the parts from the same supplier if possible (ended up using newegg.ca)

Since I already run MythTV, it was an obvious starting point and given I don’t have an affinity to a specific Linux distribution, I looked at Mythbuntu and Mythdora since I’m familiar with and already run both Ubuntu and Fedora distributions.

After downloading the Mythbuntu 10.10 ISO disk image, I discovered I didn’t have my USB DVD drive, so I wanted to create a bootable USB flash disk.  I followed the excellent instructions at https://help.ubuntu.com/community/Installation/FromUSBStick and successfully burned a bootable Mythbuntu disk on a 2GB USB flash disk via a Ubuntu VM running on my MacBook Pro.


Continue reading →

MySQL Notes

Posted on by

MySQL Command Line and Configuration Notes

Drop tables with wildcard:

There are multiple ways to specify MySQL credentials, this is not the best, but simply an example of how to drop tables using a wildcard pattern. In this case, command line history such as .bash_history will store your MySQL username and password plaintext, and an extended process listing will also reveal both username and password. When run from the command line like this, the SQL commands and the credentials are not stored in the MySQL history file (.mysql_history).  On closed (private) systems, the risk is low, especially if you clean up after these maintenance activities by purging the command histories.

mysql -u user -p password database -e "show tables" | grep "table_pattern_to_drop_" | awk '{print "drop table " $1 ";"}' | mysql -u user -p password database

Update WordPress home URL

Posted on by

There are times when moving or copying WordPress blogs from one server to another, the owner may want to update the URL associated with the specific site.

A simple MySQL update can match the WordPress blog to a new site URL:

mysql> select option_value from wp_options where option_name = 'siteurl';

+--------------------------------+
| option_value                   |
+--------------------------------+
| http://www.example.com |
+--------------------------------+
1 row in set (0.00 sec)

mysql> select option_value from wp_options where option_name = 'home';

+--------------------------------+
| option_value                   |
+--------------------------------+
| http://www.example.com |
+--------------------------------+
1 row in set (0.00 sec)

mysql> update wp_options set option_value='http://server.newsite.com' where option_name='siteurl';

Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> update wp_options set option_value='http://server.newsite.com' where option_name='home';

Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
 

Epitome of bad software

Posted on by

There is a reason many people loathe Microsoft software. Before you consider flaming me for that statement, I realize all software has flaws, bugs and eventually crashes. In my experience, even if it’s patched and up to date, the following image happens FAR too frequently with Microsoft software.
Microsoft bugs
I don’t recall having the same issues with Concept Draw, even with complex diagrams. Since I’m just tired of having to redo work over again, good-bye Visio, I’ve just purchased your replacement.

Resetting WordPress user passwords

Posted on by

Resetting WordPress 3.0 user passwords can be done directly within MySQL through the following procedure.  This assumes your installation of WordPress stores user passwords in the wp_users table as MD5 hashes and the unique site prefix for all WordPress tables in MySQL is _x.

Connect to the database via your favorite GUI (phpMyAdmin, Navicat) or command line with either the WordPress role account or any other MySQL user account with select and update privileges on the WordPress database:

update wp_x_users set user_pass = MD5('123abc890') where user_login = 'administrator';

This will update the password for user ‘administrator’ to ‘123abc890’.  Once this has completed, either flush the wp_x_users table or exit the tool used to access the database to cause the updates to be committed.  Sign into WordPress with the new password and optionally change the password via the user interface.

Apple Exemplifies Fine Software Engineering

Posted on by

So I’ve been a recent Apple user for a mere eight years, when I purchased my first iBook  running the new OS X (10.1). I’m a fan of the form engineering that goes several steps beyond the basic function engineering that is so prevalent in consumer technology these days. For Apple, it’s not good enough that there’s windows, they have to look good too – like a master craftsman that puts finishing touches on the product rather than just slapping some cheap molding on and calling it done (or Windows).

This is too fine for words.

After working through successively newer notebooks (iBook, PowerBook, MacBook Pro), I have recently upgraded my first gen MacBook Pro to a new uni-body MBP. All the way through the online store (with the complication of being a grad student and navigating the education part of the online store), the process was pretty painless. But the real wow was when my new MBP showed up three weeks ago and I decided to use the Migrate function to just suck the contents of my old MBP to my shiny new uni-body MBP (thanks for the encouragement, Jonathan). I figured since I didn’t have the time or energy to setup another computer from scratch, I would try this migrate feature – with a heavy dose of battle earned skepticism. When I turned on the power on my new MBP, it seamlessly guided me through the setup .. and asked me if I wanted to migrate from an existing Mac or even a TimeMachine backup of a Mac.  I said yes, hooked the old and the new together .. fully expecting this to not end well and have to restart some install process.  Well a little while later, the migrate was done .. I restarted my new MBP (didn’t have to), and it looked exactly like my old MBP. All of my Applications were there. All my documents where there. iTunes was there. iPhoto was there. The positioning of the icons and documents on my desktop was exactly like my old MBP. Wow. A migrate function that actually worked.  Really. All the way.  Ok, well I did have to re-setup my home wireless connection .. for some reason that didn’t seem to come across, but with the totally customized settings I use, I’m not too surprised although it only added about 120 seconds onto my migrate time.

So at the time I’m writing this, Apple has announced the next generation of the MacBook Pro (the Intel i5 and i7 processors).  Since I’ve only had my shiny new uni-body MBP for a week, I call the folks at Apple and speak to a very pleasant customer service rep (send me an email or website message and I’ll forward his name), who not only cheerfully agrees to accept my new MBP back, but helps me order the new generation. They waived the return shipping and any refurbishment fees, as well as the express shipping for the new unit to me.  Gives me his direct line so if the Apple provided UPS return sticker expires before I get the old-new MBP migrated to the new-new MBP, I can call and get a new label. All this (and I ordered a new mouse) and they refunded a net of nearly $900 back to my credit card.

Well, I’ve just finished the migrate from the old-new MBP to my new-new MBP and again, it was seamless. I don’t think I’ll rebuild a new Mac from scratch any more – this is just too fine for words.  So I can get back to my Master’s thesis and life in general, and not worry about the software out there that is half baked or just barely good enough to get by .. with lots of manual care and feeding.

Thanks Steve and crew – this is why I’m an Apple shareholder.

Accessing Ubuntu desktop from Mac Snow Leopard

Posted on by

Accessing my Ubuntu 9.04 Gnome desktop from the built in Mac OS X 10.6.2 VNC viewer took a bit of tweaking on the Ubuntu Gnome side. I have an OpenVPN SSL tunnel between the Mac and the Ubuntu desktop, however a SSH tunnel could also be used to protect the VNC session. In this post, I’ll just cover the VNC server setup assuming a secure connection between the Mac and the desktop.

Initially I followed the guidance at sanity, inc.”How to OS X Leopard Screen Sharing with Linux“, on Ubuntu I installed tightvnc:

apt-get install tightvncserver

Then tested it out by starting up the vnc server on the Ubuntu system as the user I want to run the remote session as:

tightvncserver -geometry 1024x700 -depth 24 :1

As tightvncserver starts up the VNC service, it will check for a .vncpasswd file in the user home directory. If it doesn’t exist, you will be prompted for a password to use to protect the remote session.  Note VNC is not designed to be used for multi-user remote access.
On the Mac, rather than use Bonjour to automatically discover the Ubuntu screen sharing service, I just referred to the VNC session directly within Finder which invokes the built in VNC viewer. Enter the VNC session password when prompted and the Ubuntu desktop is displayed. connect-to-server Within Finder, either use Go -> Connect to Server or Apple-K to bring up the Connect to Server window.  The server address is the URL that points to the Ubuntu VNC instance vnc://10.10.1.2:5901 where the port is 5900 + the display number specified when starting up the tightvncserver (5901).

This all worked fantastic, except for the keyboard mapping within Gnome – it was scrambled.  After googling several possible solutions, the only one that was successful for me was to disable the keyboard plugin in Gnome

Amit Gurdasani wrote on 2008-04-28: #51

I’ve also encountered this issue with TightVNC and the hardy release. My solution was to capture the xmodmap -pke output as ${HOME}/.Xmodmap at the login screen (DISPLAY=:0 XAUTHORITY=/var/lib/gdm/:0.Xauth sudo xmodmap -pke > ${HOME}/.Xmodmap). When gnome-settings-daemon starts up and finds an .Xmodmap, it asks if it should be loaded — I answer yes. As a side effect, if gnome-settings-daemon were to be restarted without the .Xmodmap, it’d scramble the keyboard layout again. With an .Xmodmap in place, it’ll load the .Xmodmap every time.

Due to another issue (#199245, gnome-settings-daemon crashing with BadWindow every time a window is mapped), I disabled the keyboard plugin using gconf-editor, at /apps/gnome_settings_daemon/plugins/keyboard. Since it’s not being loaded, I suspect it might not garble the layout even if I remove the .Xmodmap now.

So maybe disabling the keyboard plugin is a better fix.

On the Ubuntu system, invoke the Gnome configuration editor (gconf-editor on command line), then navigate to apps -> gnome_settings_daemon -> plugins -> keyboard uncheck the Active keyword.  Kill the VNC daemon and relaunch it – problem fixed.

pkill vnc
tightvncserver -geometry 1024x700 -depth 24 :1

Various methods exist to automatically start and kill the VNC server, but for now this will do it for me.