japester: (Default)
I've been digging into performance trends and metrics over the last few months and trying to find a sane and scalable app for deploying across Unix space. I'm really saddened by the complete and utter lack of useful open source software in this space. There are some of the semi commercial apps which are getting there, but are somewhat difficult to get running on Solaris. The GNU/libre ones are either complete and utter tripe, or suffer serious architectural faults. (And yes, I do suffer from the 'If it doesn't work as described by the documentation, I'll throw it away. Documentation is not hard people)

  • Hyperic.
    bloated and awkward interface. graphs? um .... they're there, somewhere. Damned if I could work out how to get it do them easily
    Summary: overweight monitoring app that doesn't do performance metrics
  • munin
    Just works! I am very sad to say that it gathers data by pulling it from hosts, that's a method that just does not scale.
    Summary: Very easy to use. works well in small/medium environments. Will not scale.
  • collectd
    collects data well, but there is no frontend worth using. They are all, and I mean all, alpha quality. Heymon appears to have come closest to being able to 'actually work,' but it's pretty ugly still.
    Summary: Reasonably easy to get items monitored and configured. Build a working and stable front end and it'd be just right.
  • hobbit/xymon
    Also, just works. It has some serious architecture flaws though, like trying to navigate just which file to edit to get it to monitor something, and problems with how it generates rrds.
    Summary: flawed architecture

I'll be looking at others in the near future. if you have any good/bright recommendations, I'm all ears. I'm not fussed about alerts/alarms, just performance/utilisation metrics.

Oh wait, the other rant.
Why the f*ck is it so damn hard to build anything on Solaris? I am obviously spoilt by having had the dream life living with Debian (and Ubuntu), where if i want to test some random app, all I have to do is 'apt-get install widget' and voila! It's there. and it runs. If it's not quite what i need, apt-get remove widget, and it's gone. Try that on Solaris? OpenCSW come kind of close, but does not have anywhere near the breadth of app choice that Debian has. Oh, and people don't build stuff on Solaris, so it's fairly common to find that it just won't work.

To get over the 'problems getting stuff built on Solaris', I've spun up a couple of VirtualBox Ubuntu servers and have been developing on them, then porting stuff back to Solaris. That's kept the frustration levels reasonably low.
japester: (Default)
I've been spending a fair chunk of the last month getting my brain wrapped around Reductive Lab's Puppet product. It's a configuration management framework aimed at SysAdmins who spend more than 5 minutes a week managing config files.

It has taken me quite some time to actually get started on it, as the learning curve is quite steep. The online documentation is enough to get you started, and then missed significant steps on getting from basic configuration to a usable environment. So, it's a typical OpenSource product then. I'm going to write up my experiences with getting started sometime RSN.

It's good. It works. It's frickin' powerful. So much so, that one of my colleagues is too scared to touch it, he believes that it's too easy to wreck your entire environment in quick easy steps. Well, that's half the fun, really :)

I have sucked in all our core infrastructure to the puppet brethren. Over the coming days, or weeks, I will roll the rest of the application servers in as well.

The only point of contention and definite confusion will be to know what is managed and what is not. I can see that being painful for the NOC monkeys who come in in a month (or three) to manage this environment. There are 90 of the little Slowaris beasts here though, so having some form of centralised management is required. Otherwise entropy will develop at a rate too quick to manage.
Just adding in hosts keeps me on my toes as each one has so far, at least one thing different from the SOE that it should be conforming to!
japester: (cathaiku)
or .. didn't the French cop a whooping at Agincourt?

nah .... there's some hilarity ensuing.
Newcastle is under a layer of hail at the moment. Take a squiz the radar image )

The funny bit, one of our clients is up there. (think back a few weeks ...) and they called a little while ago, saying, "please shut down the server, there's a serious storm up here and our building is flooding.
Okay, connect to server start to shut it down, while on the phone to one of the dudes up there and the phone drops out, along with my connection to the server. oh, no phones. no phone line. no internet. Get a mobile up there and walk them through shutting the thing down and disconnecting the batteries in the UPS.
why go that far? There's water running down the wiring loom directly into the rack!!!! Batteries + water == much badness. Generally b00m!-ey badness.
Thankfully, the guys up there are relatively clueful. They understand that water and electronics mix and produce magic blue smoke.

Still amusing
japester: (user error)
There was a site visit I did on Monday, that was so stupid, I am glad we charge an arm and a leg for my services.
One of our favourite clients - favourite in the non sarcastic sense, they have an office in the Gateway building right next to Circular Quey, called us up at 1230 to say their network was down. They have a funky network. We are the hands on people who do onsite legwork for the (stupid) IT cowboys working in Hong Kong, with overall (clueful) management based in London. At 1230, London is still asleep and will be for some hours to come.

One of the firewalls had decided to play up. HKIT (HongKongIT) requested one of us to go onsite and diagnose, and if necessary do whatever was needed to bring the network and VPN back up. We do not have a login onto the box (with good reason!) and neither did the HKIT droids. Only the guys in London know the root password for the firewalls. Given the past inexpertise demonstrated by the cowboys, this is a good thing (Tm).
Most of the time.
Some hour and a bit later, we pulled the power lead out of hte back of the firewall that was playing up. HKIT got drilled for doing this not so long ago but at 1400 Sydney time, and London at 0400, it was at least 4 hours before we'd be seeing them again and 3 hours of network outage is bad for a company that lives on email.
Thankfully, it came back up cleanly. Yay for a variant of some randon Unix running checkpoint.
Yay, that's 2 hours of work + travel time.
Bizarrely, it took me 50 minutes to get from Circular Quay to St Leonards, a trip that if the trains sync takes 30 minutes. The trains did not sync well at all this time. So .... yay 3 hours of chargable time!

Out of all of that, I power cycled a firewall. Call me a pushbutton monkey. Whoop de dooo.
japester: (dont make me come over there)
interesting snippet for those who have been watching the job status at my work.

after being poked by myself and a few rectruiting mobs, the job description for our new lacky has been updated. It reads less like a 'senior sysadmin' and more like a junior. maybe we'll get some applicants now :)
japester: (penguinmashingkeyboard)
For reasons only known to management, we recently bought a shiney new Dell PowerEdge 1950.
it has blah blah stastics and the main reason it seems that we did not get a Sun X4100 was the sun sales/marketing droids were on a holiday with being reasonable.
so ...
software compatability. It doesn't like running debian. no network card or raid card detection.
well. I have a fully functional heater!

It'd be a bit easier if it wasn't genuine new hardware, as I'd be able to pick someone else's brains. as it is, i've not been able to find anyone else's install hints or ... anything!

japester: (answer)
The dudes we interviewed last week were, shall we say, not appropriate.

My boss just said, and I quote, "I'm going to a random person off the street and give them a job."

le sigh. Why is it so hard to find good people these days?
japester: (Default)
if you're in sydney and feel like coming to join me and work with a vibrant company inhabited completely with geeks (girl geeks too), we need one. or possibly even two.

description here. and it has come to note, that people in the past have not read the instructions on this page, and they get laughed at.
we're a very funny bunch here.
japester: (bitchsmite)
I just lost 2 hours of my life to tinydns. I swear, if ever I see djb, I will punch him in the face.
His software, while security conscious and stable, is so freakingly uncformist to standard un*x configuration practices that troubleshooting it is like trying to extract the one non poisoinous snake from a pit of ravenous terminally scary snakes. While blindfolded. and one arm behind your back.

it's not helping Friday night come any closer either.
japester: (Default)
I've just been asked by my place of gainful employment if I am interested in becoming our in house MS SQL guru.
It will involve being trained and possibly certified, with the aim of supporting one of our major clients 24/7, on call too.
I'm intrigued, yet not completely sold on the idea.

I have an interest in DBA stuff, however historically, I've been more interested in Postgres, a little less so in MySQL and not at all in MS SQL. Would adding this third set of knowledge be good or bad for my interests and abilities in the other two. Or would it add to it? The aim is to be a DBA admin, not a programmer.

In conflict with the time quota required for this skilling, is my dedication to wanting Solaris and Cisco training that I've been pursuing in my own time. Will this extra load be too much to learn in a short time frame?

I'm hesitant on a couple of levels but the more I think about it, the more useful I can see it being.
I throw this to the wider, and possibly more knowledgable crowd.

Do I go for this or not?

(X-posted blatantly everywhere)
japester: (bitchsmite)

been here almost 7 hours. working straight through.


much more of this and I'll be bringing in the hammer from the car
japester: (Default)
NG's on a roll again.

Roses are red,
Violets are purple,
Which is a very hard word to rhyme
And makes me happy that on February the 14th we don't traditionally have to give each other oranges.

Except a certain Tom Lehrer took exception to that.

"Eating an orange
While making love
Makes for bizarre enj-
oyment thereof."


Today has been ... interesting. Yesterday was another Monday. Today, I went for a wander down to Arts @ UWA to look at their shiney Dell racks. hmmmm looking at racks hmmmmm and they are very nifty and extremely feature-full for their very low low price - about a third of what we paid for our previous crappy ones.
I got mistaken for my previous IT Manager yesterday. Admittedly, by a marketing dude, but it was still amusing. Yesterday I also wielded the LART of extreme doom. It's what you get for doing silly things like account sharing.


Dec. 7th, 2005 12:33 pm
japester: (draco inna dress)
1249 and i'm still asleep.

just call me the afternoon shift.
japester: (chaos inside)
into me. or at least, starting to feel like it is :)
a few things happened today that made me feel like i actually am becoming the Responsible One (Tm) at work.
having a hard drive fail on one of our SANs and poking Sun about getting it replaced. (yay gold maintenance)
watching the SSL certs expire on two servers and pulling our hair about updating them (remember, we hates Eudora, not imapd)
and applying to UCS @ UWA to become the subnet administrator and security contact.

yesterday, poking the firewall software into my brain, and wrapping my head around ingres did nothing for those feelings. maybe because I should already know them ...

My patience will certainly miss [livejournal.com profile] kasinik when he is gone.

It has been happily noticed that I've been arriving at o'God o'clock for the last fortnight. 7:30 is *not* my normal start time. it's a very scary time to be at work. but it does mean I can leave early. i should start doing that. wednesdays are the perfect candidate, so I can get to training early too. miracles? performed on alternate days.
japester: (Default)
You Are A: Monkey!

monkeyMonkeys are intelligent and agile, well-adapted for jungle life as they swing happily from tree to tree. As a monkey, you are a social animal who prefers a warm climate, eats a wide range of food and is quick to learn new things. A monkey's tiny primate features are irresistable, as is his gregarious personality!

You were almost a: Mouse or a Duck
You are least like a: Groundhog or a PonyDiscover What Cute Animal You Are!

yep, haven't changed a bit.

went down to UWA for some work discussions with the Arts sysadmins. (sorry [livejournal.com profile] bondles, if i'd not have a meeting to get back to, I'd have dropped in). t'was quite productive. looks like we can definitely do some infrastructure and knowledge sharing. I *finally* managed to cajole james into giving me some of their maintenaince scripts so I can better integrate my client network into a database - and then let the other guys here give people software, rather than it always falling back to me.

and soon, i'll be off for a drive to nedlands to go plug the director's printer back in.


japester: (Default)

August 2013

18 192021222324


RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags