Archive for the 'System Administration' Category

Linux Recruting

I get a LOT of emails from headhunters, many asking me to come work for them doing every-damn-thing for no money as a “consultant” on (only) a 6 month contract :) I am sure all tech people do. Occasionally I get email from a recruiter who is actually asking me for help looking for a decent Linux person. I got one of those this afternoon. In summary, the email went like this:

I am looking for (Linux Admin) and you probably aren’t looking but I am having a hard time and could you help point me somewhere I can find one?

I always respond to those emails, and, for posterity and for any recruiters watching, here’s the answer:

Not necessarily true. I am always looking ;)
I get a lot of requests and offers and I’ll tell you what turns me off and that may help you find someone. Linux guys with any experience are in really short supply and they are a unique breed of techie. Most are driven to Linux by the premise of free software and/or open source ideals, and as such they do not necessarily have (current) windows skills and are even more likely to not be interested in using any that they do have. I fit into that category. Also, not every Linux guy is a java programmer/desktop technician/helpdesk/printer mechanic/insert other required skill set jumble here. I see a lot of those. “We need a Linux guy that will fix our windows desktop, program new device drivers, fix our mainframe and telephone system, sweep floors and wash cars” kind of things. Those kind of people do not exist :) Lastly is the compensation. Most companies have dealt with the influx of paper certed, dime a dozen MCSE’s for their technical needs and they truly believe that anyone out of grade school can “do tech” for them. It has greatly devalued the industry as a whole. They do not understand the real high skilled people are rare and expensive and can *easily* find work, which is why most Linux/Unix people have not been effected by the technical recession.

So I guess in short,
Linux guys are almost always staunch Linux guys (and if they are not, be suspicious).
Be specific in what you need but remember that These kinds of tech guys are quick at catching on to related technologies so try and be general where you can. For example, there are a bunch of scripting languages and all of them are capable of getting the job done, so say you need a scripter instead of you need a perl scripter.
Be prepared to offer more compensation for a rarer Linux tech than you would an unemployed Windows tech.
Advertise in the right circles. When I get offers, I often send them out to some of the mailing lists of Linux techs I am on, and there are some great Linux groups on Facebook and Google Plus. There are also websites like Linuxquestions.org where Linux geeks hang out.
Lastly, if all else fails, try a few less experienced Linux guys.

Advanced Programming in the Unix Environment 3/ed

Good gracious this is a big book! What’s funny is I KNOW I have read and reviewed a previous edition of this book and I spent a half an hour looking for it this morning, but it must have been before I moved and on my old Blog. That being the case, well it’s high time you heard about this monster!

This book, Advanced Programming in the Unix Environment, by Stevens and Rago, is the 3rd edition of what is, essentially, the Unix Programming Bible. In fact, so much so that I cannot imagine any serious Unix/Linux/**ux contributor that doesn’t own a copy or at least know what it is.

This is *not* light reading. It is a reference book. This is the stuff geek dreams are coded in and you are going to want to be familiar with the C language to get a lot of this.

All the internal workings and ideas about this kind of operating system, how it works, or is supposed to work and code examples are included here. The least technical chapter in here is the 1st, which is the overview chapter. This goes over things like input/output, files/directories, processes, error handling, and system calls. From there, the chapters narrow in more on specific subjects like Process control, Daemons, Signals, Threading, etc.. Like I said, there is a LOT of very specific information in here. That being said, if you are developing anything more than some scripting, this has what you want to know. This is not to say that those are the only folks that can get anything out of this book, though. Even without understanding the code examples, a person could get a good understanding and overview of how this fantastic type of operating system works, and why. This is the category I find myself in more than any other. Although I have done some C programming, I find myself using this book to help me conceptualize how things are working the background.

No self respecting Unix/Linux geek should be without this book in one format or another. The hard copy I have was sent to me by Pearson Education for the purpose of review. They sell this in book in dead tree format for $70 and $45 for the electronic version. That may sound like a far bit of money, however, remember this is not a story book you read once, this is going to be something you turn to for the right information when you need it. I almost always give away my review books after I read through them, but this one is sticking around. In fact, I am just going to take it to work with me so I can have it handy where I would normally need the information anyway.

BackTrack 5 Cookbook: Quick answers to common problems

BackTrack 5 Cookbook

BackTrack 5 Cookbook

You know, sometimes, just sometimes something fortuitous happens to me. This was one of those times.

I was contacted by my friends over at Pakt Publishing to review their new book on BackTrack. Of course I said sure. Hey, I am a Linux junkie after all! It had actually been quite a while since I had played with BackTrack and this gave me *just* the incentive I needed, but let me tell you a bit about the book…

The book is a “cookbook” style book which gives you “recipes” or guided examples of common problems/scenarios and their fixes. The book is well written, a good reference for a pro, and a great tutorial for the beginner, and by beginner I am assuming that the person *does* have Linux experience, just not BackTrack experience as some command line comfort is pretty much a necessity for this kind of work. The first 2 chapters start you out exactly the way they should, by installing and customizing the distribution. What they don’t tell you is it takes a good while to actually download the distro, but that is beside the point.

Once you actually get things running well, you can follow the book through some really decent examples from Information Gathering all the way through Forensics. The book covers all matter of subject matter and applications in between such as using NMAP, Nessus, Metaspolit, UCSniff and more. I mentioned that this was fortuitous for me and that was because one of the things the book covered was the Hydra program, and, as it turns out, that was the perfect tool for me to use in remediating some password synchronization issues across several hundred servers.

Anyone using a computer should have at least a basic understanding about keeping their valuable data safe, whether that data is for a multi-million dollar company or your own invaluable family photographs. This book goes to great efforts to not only explain how to detect, analyze and remedy such issues, but also gives important background about just how systems become vulnerable to begin with. If only for that reason alone, it’s worth the read. If you are actually a sysadmin, this information is a must. For $23 for the ebook version, it’s a no brainer. Good book. It helped me out and I’ll wager that if you give it a read it’ll do the same for you!

Screenshots

I have long been fascinated by different peoples computing environments. Somehow I believe it shows a little glimpse into someone’s mind. With that in mind, I thought it might be interesting to other people as well so I polled a group of my friends who are some of the most influential computing buddies I have. Here is what they sent:

Name: A.W.
What do you do?:
I’m a NetApp Wrangler and Windows Sysadmin by trade. Looking to add storage admin as well (EMC/Cisco).
Tell me about your DE?:
My main workstation is my MacBook. I identify with this machine the most and my desktop environments tend to show my personality and style choices. I like IBM style green on black terminals which I have been addicted to ever since I installed my first AIX machine (a POWERStation 320 that I got for free from my ex-girlfriend’s office). The desktop is a stylized Sylvanas the Banshee Queen of the Undead from World of Warcraft. I don’t currently play the game but I’m into zombies and undead stuff as art and game play (and hot pale powerful gothy women). My Windows 7 machine is a gaming machine and also used to do my work as it’s the best machine to log into our VPN with. It’s an Alienware with the Phobos Red theme and the LEDs are currently all set to red with a pulsating skull on the front. It’s kind of Darth Vader. Alienware does nice themes and some of the nicest pre-installs I’ve ever seen (yes, the first time I didn’t wipe the OS that came with the system)… It has no shovelware. I’ve owned the Powermac G5 Quad for years and bought it to be the last and best PowerPC machine. Eventually I was no longer using it as I supplanted its use with my MBP which I can carry all over the house and use wireless N with. Wanting to breathe new life into it, it became a PPC Linux test box and I’ve found the best environment with Fedora Core 17 Beefy Miracle. I’ve replaced the desktop graphic with something nicer than the default fireworks that is still Fedora themed. The Firefox window is a shot of my home file server control panel. It’s a red aluminum cased custom AMD A4 build with 8 GB of RAM, 6 x 2TB Seagates (SATA3)ZFS RAID6 and a memory stick to hold FREENAS 8.0.4 x64 MULTIMEDIA. Since it’s red I named it after my favorite Motts discontinued beverage: Beefamato.
aw2
aw3
aw1

Name: D.C.
What do you do?:
Programmer and professional Bearded Curmudgeon.
Tell me about your DE?:
vim is my IDE, and I have a window open full screen, split into up to eight or so buffers on my main screen. On a second screen I have terminals for running my code’s tests, viewing logs, and for talking to colleagues who work all over the world – my team is split between Utah, the UK, Moscow, and anywhere else that we can find good people. My windows are all slightly transparent when inactive, as it makes it easier to find stuff if I can see it when it’s behind something else. I do, of course, use focus-follows-pointer and click to bring to front, but almost all my navigation is via the keyboard. When I do need to move the pointer,I use a trackball. Desktop? Yeah, there’s one under there somewhere, but I hardly ever see it. It’s a plain neutral colour with no icons on it so it doesn’t interfere with window transparency.
dc

Name: J.B.
What do you do?:
Senior Software Engineer working on cloud managed digital media systems for the retail environment.
Tell me about your DE?:
Windows 7. I run Linux on my desktop, but I never felt like having the distribution to work to change what’s on my laptop, and I use the laptop the vast majority of the time.
jb

Name: J.F.
What do you do?:
Solutions Architect, Enterprise Services, HP.
Tell me about your DE?:
I alternate between a black desktop and this photo of my favorite car. A friend collects vintage gas station equipment and provided the setting when I took this picture. I try to keep my desktop clean and maintain a folder called “desktop-stuff” for all the junk that would normally accumulate.
jf

Name: J.S.
What do you do?:
Retired network engineer now part time Asterisk/VOIP and wireless consultant.
Tell me about your DE?:
Windows 7 for the most part, but I have a Ubuntu 12 VM running X11RDP so I use Remote Desktop rather than VNC. That’s where I do the majority of my compiling & code editing in Xemacs.
js

Name: K.H.
What do you do?:
I’m a senior engineer on the Enterprise Infrastructure Team for a state government. I wrangle Tivoli Storage Manager, VMWARE, DNS, Linux/Apache/MySQL/PHP, legacy and modern UNIX/Linux, SANs, some LAN/WAN, provide support to the CISO in all areas of infosec as needed, and function as troubleshooter of last resort for any given problem.
Tell me about your DE?:
Windows 7 would not be my first choice, but since I have to use Windows-only apps in the execution of my duties, it is the best for the job. I run two monitors, which have different resolutions, but this is the best that can be managed on a restricted budget. Ideally there would be two 23″ monitors, but if we’re dealing in ideals, I would have an Alienware laptop instead of a Dell. The theme is a transparent space-based theme courtesy of NASA, but the background is an image from Stickman featuring some of my favorite tools. Rather than hide the start bar, I leave it up all the time for quick access.
kh

Name: L.F.
What do you do?:
I.T.Manager and Senior Linux Admin, LAMP developer, scripter and all miscellaneous duties as assigned.
Tell me about your DE?:
Mint #newest_version running my usual slew of apps and xterms on 2 dual monitor machines. Dark wallpaper is currently a “black leather”. I like dark unobtrusive wallpapers best to avoid distraction. Windows running in a vm, where it belongs. Just can’t have enough desktop real estate you know! And, yes, that’s mutt for email – best client out there.
lf

Name: M.H.
What do you do?:
I’m an I/T support specialist and dispatcher.
Tell me about your DE?:
I have quite a number of different desktops really. In fact I always have had. When they get cluttered I throw things into folders and eventually archive them if I don’t want to delete them. (My folder structures in my home directories is horrible.) Each system I use has a different purpose. The desktop here is my home daily driver. Multiple screens often dictate what wallpaper I use though frustratingly it’s hard to span wallpaper across multiple monitors. At home I usually use single displays but at the office I use four screens total. Working on adding another one. ;-) As for colors I prefer a darker theme with light lettering. For terminals I prefer a black background with amber text or as close as I can get using a color picker. Green if I don’t have amber as a choice. Translucent terminals look nice initially but are a pain for me to focus on.
mh

PHP and stuff

Lately I have been working so hard that I haven’t even had any desire to do any fun computering at home. Today that changed a bit.

I decided this morning that it was high time I upgraded my all time favorite rss feed reader, tiny tiny rss. Well, wouldn’t you know it, after I did the install I found it required a version of php higher than I had available on my server. Time to upgrade.

I run Centos 5 on my main server and, by default, that carries a php 5.1.x. I needed 5.2 or greater. As it happens, php 5.3 is available in the repos, so I did the upgrade. For the uninitiated, that entails doing a “yum list installed | grep php”, which gives you a list of what you *have* installed. Next you remove php by doing “yum remove <and name all the packages in the prior list here>”. This, followed by “yum install <list of files for php 5.3>”. For example, I had php-common.i386 and php.i386 installed, so I did a “yum remove php-common php” and then “yum install php53-common php53″ to get all my php 5.3 packages on there. This was followed by a quick “service httpd restart” to make sure my webserver was using the new version.

Murphy’s law states that “something will go wrong if it can”. Well, *MY* law states that “something will go wrong”, and it did. As it turns out, I had built a whole bunch of php applications maybe 7 years ago that my wife uses almost daily. In the olden days of php, you could declare a php script at the top by doing a “<?”. NOW, you need to declare it by doing “<?php”. Consequently, nothing I had written worked. It only took me a minute or two to identify why the problem was occurring, but fixing it was another story.

So, how do you find all the files you have to fix? Well, I used the “grep” command. More specifically, egrep. I went to my html root directory and searched by doing “egrep -r “<\?” * | egrep -vi “<\?php” | egrep -vi “<\?xml” | grep -v inary”. What does all that do? The first stanza looks recursively through the directory structure at every file and outputs the ones that have any “<?”‘s in them. The second takes that output but does NOT pass through any that are “<?php”. Why, because they would already be ok! The third takes the results and doesn’t pass through any that contain “<?xml”. The last one doesn’t pass through results from binary files. The end result is I had a list of directory / file / line information of all the files I had to change / update. A few minutes later, after using vim, the best text editor around, I was back up and running!

My aching butt? No more!

Knoll Generation

Knoll Generation


I have long held the opinion that you should not skimp on the things that you use professionally and frequently. For instance I often tell my peers to make sure they buy decent computers and not just bargain basement models. For some reason it never occurred to me to apply this reasoning to my posterior. That is, until recently.

I have been using a most uncomfortable office chair for the longest time, in fact, for the last 5 years. It’s one of those “conference room” models, which, although built plenty rugged, are not necessarily built for comfort. The real problem with this is I sit in the thing almost 8 hours a day. I even had to buy a pillow for it because I have what I like to refer to as “sysadmin’s butt”, which is essentially the lack of a posterior portion of your anatomy, due mostly to parking it in a chair 8 hours a day for many years.

A few days ago, a friend of mine who just happens to work for a great company called Knoll, got me hooked up with a review unit of the Knoll Generation chair. Let me just say I don’t know why I waited so long for a good chair!

This chair is the epitome of office comfort and has so many options to aid you in that regard it’s difficult to mention them all. Of course it offers the standard amenities such as height adjustment and a reclining back, but wait, there is more! It has these cool adjustable arms that not only raise and lower but they also have arm pads that adjust in depth, width and pivot. The back of the seat has this comfort type netting which is flexible, cool and very comfortable. The top of the back flexes almost over backwards and it a great place to rest your arm while sitting sideways in high comfort. I think the feature that threw me the most is the flex seat. Unlike most chairs with a solid unyielding seat, this seat has some degree of movement or side to side pivot. I almost thought that my chair was broken until I realized that this was intentional! This pivot to the seat allows comfort and support for how people sit. What I mean by that is people sit different ways and on different angles with their feet up, legs crossed and what have you, and this seat allows some motion in that regard to keep even support and pressure where it should be, fully on your posterior instead of on your legs when you are not seated “flat”. In addition to that, the seat depth adjusts as well.

Although this chair may be on the expensive side, I believe it’s well worth it, especially for people who are confined to such devices for much of their day. I wish they had a travel version available, I would buy it in a minute ;) I think the best testimony is that every person in my office has now sampled the chair and ALL of them want one! If you are in need, you can’t go wrong with one of these. It’d most likely be the last office chair you would ever need to buy.

My only hope and wish is that the Knoll folks see this review and decide to send me one of their other products to test as well! Other than that, I’ll be happy to hang out in my own Genertion!

Lost your Mint password?

First time this happened! A coworker asked me today how to get into his Linux Mint box after he forgot his password. Of course I rattled off the old GRUB way to get things done, but, what?? This is GRUB 2! No so fast there! Turns out it’s quite different.

You hold down the shift key while booting to get to the grub menu.
You hit ‘e’ to edit your boot options.
You change the kernel line options on the very end of the kernel line to read “rw init=/bin/bash”.
You press F10 to boot.

Once booted you are dropped immediately into a shell prompt where you can change your password with the “passwd username” command. Reboot and you’re home free!

Rsync bug

rsync

rsync


Bitten by the rsync bug? I was. Apparently in the new RHEL 5.7, and I am sure the RH clones like CentOS, Scientific Linux and ClearOS(?) as well, there is a bug in rsync when you use it with ssh transport like so:

rsync -avz -e ssh remotehost:/data /data

The fix is to make sure to append a username to your host and then it magically starts working properly again.

rsync -avz -e ssh username@remotehost:/data /data

Enjoy!

Mint 11 Boot Splash

I have seen a lot of commenting on the decision to use a black screen as the boot splash on Linux Mint 11. One person summed it up well when they said that that black screen is indicative in many other operating systems as something going wrong and it tends to scare people. Well, that being the case, if you are scared or if you just like to see what’s going on behind the scenes like I do sometimes this is how you can fix that fairly easily.

Open up a terminal and do a:
vi /etc/defaults/grub

scroll down to the line that says:
GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash”
and change it to:
GRUB_CMDLINE_LINUX_DEFAULT=”"
and save the file.

After that do a:
update-grub

When that is finished, reboot your machine and enjoy watching the text based boot process as it occurs. Pay close attention, though, ’cause it sure doesn’t last long! ;)

Linux Shell Scripting Cookbook

Linux Shell Scripting Cookbook

Linux Shell Scripting Cookbook


   As a full time Senior Linux System Administrator in real life I was quite interested to get my fingers on this book for a review. After all, the job of a smart sysadmin pretty much dictates scripting away as much of your work as possible. We are a lazy bunch and we call that being efficient :)

   This is the first book I have reviewed by Packt Publishing or the author, Sarath Lackshman, I wasn’t really sure what I was in for. In fact I was slightly put off by the price, which I initially thought overly hefty at $45 US. For that kind of scratch I am used to seeing a much more substantial sized book from the sort of publishers I normally review for. I started making my way through the book anyway, and I am glad I did.

   What makes this book really cool is the premise behind it. Inside, as a “cookbook” should, you have these “recipes” for scripts. These are not what I have normally seen in many scripting books before, which are generally theoretical and sometimes lengthy examples, but these recipes are pretty straight forward, real world examples of things you might want to do, and how to handle those efficiently. The recipes are also small enough that you could easily piece meal things out to compose another script and I am certain that would be a great help to novice scripters.

   As nice as I think this book would be for novice scripters, there is a lot of smart stuff in there, stuff that had never occurred to me through my years of command line use. I actually got really excited to try some of the examples in there and to put them into practice. I particularly liked the little tricks here and there, like the “subshell trick” and I was absolutely thrilled that this book used modern syntax and variable manipulation, dropping the deprecated stuff like putting commands into back ticks. Good form!

   This book is certainly a keeper and I would recommend it highly to anyone who wants to become proficient on the command line. Some days you actually *do* get what you pay for, and I believe people will find this book to be a good example of that. This book was truly fun for me to work my way through and I sure hope they have more like it in store for the future. Go buy yourself a copy. I know I will be hanging on to this one for a while :)

Next Page »