The Industry of the Rhaven

adch++ for Arch Linux updated

2012-01-28T00:45:00.000-05:00

So I maintain a package for ArchLinux that beats the adch++ into a unix environ. Its a hubsoft for the Advanced Direct Connect system.

If you haven't heard of ADC, its a file sharing network based around the idea of centralized message passing 'hubs' very similar to IRC. The clients connect to this hub for chat and search. Searches are performed by clients sending a request for X to the hub, and the hub repeating it to all participating clients. Who the search/chat/requests are sent to can be controlled by the hub. Each client then searches its own file list (distributed computing!) and returns the results directly to the initially searching client.

This was ripe for abuse so the hub has protections to stop the DDoS attacks that can be caused by searches. Most clients and servers implement TLS certificates for encryption as well.

I like adch++, and use EiskaltDC++ (gtk and qt version) as my client.

Why Walls Suck

2012-01-16T21:55:00.002-05:00

I'll try make this short. As a member of a hackerspace, and particularly one in a urban area, space is a problem. We always need more space, and we are continually getting more things to fill that space. We put things in corners, find inventive storage solutions, stick things to I-beams with hard drive magnets. And it still isn't enough.

Yes, that is a 24 port switch stuck to an I-beam.

Bringing me to walls. Walls are the most basic way of designing a space, cutting order and purpose from the blankness of a single large room. While personally I prefer a large blank space to live in, a hackerspace requires delineation. Some people want to socialize, others want to spray 20 foot gouts of sparks as they metalwork, and others need to spread out delicate fabric to make a new LED gown. When we make our walls though, we sacrifice the space under them, around them, behind the door, above them. All of it, lost to a giant piece of framed 2x4s and drywall. So we attempt to salvage some of that wasted space by running infrastructure that everyone needs through it, under it, on it. Power, network, video, racks, shelves, art hanging, coat hooks, magnet strips and a thousand other ways we try to reclaim the space that the wall eats up.

Bringing me to how walls fucking suck. You need to find a stud behind the drywall, either drill a pilot hole or smash a barbaric nail into it, praying you have it lined up correctly and that you didn't just split your board in half. Then when you want to change something, dry wall crumbles, needs puttied and sanded and repainted. And even when you have your fancy screw and your rarbarbarbar nails, how much can they hold? A medium wood framed painting? A light spring jacket? I'm not asking to make a non-load bearing wall perform miracles, but this is just pathetic.

But conduit you say! Properly laid out it lets avoid the walls. But have you ever tried to pull a piece of romex through 1 inch conduit that already has something in it? Pain. In. The. Ass. And I ask you, what was the last building you were in where power was an afterthought? Where low-voltage comms cable was an afterthought? Power. Data. Environmental (Heat/Cold.) Water. Waste. These are basic functions of any 'building.' If I gave you a 5000 square foot square building with just walls, you would tell me its not finished. I give you a 100 square feet room made of cardboard boxes with full cable TV, Wifi, phone, electricity, plumbing and lights, and you know right now in your head you're thinking you could make do. This is what I'm talking about.

Useless piece of drywall.

We need to think about walls for they were intended for. Logical space separation, sound isolation, privacy. These things are not more or less important that modern infrastructure. I'm not asking that people change the way they live; picking engineering efficiency over human nature is a demonstrably bad idea. Spaces must be designed with their living occupants needs above all others, but as engineers, as contractors and technicians and all the rest who are told to 'make stuff work' we need to build a better paradigm for walls. We can do better.

Bandwidth.com, still the worst ISP ever.

2012-01-09T11:19:00.001-05:00

If you read my previous post about Bandwidth.com, you can safely assume we canceled. The rest of the story:

After sending in a disconnect notice, once a week for 3 weeks, we received confirmation that we would only be allowed to cancel if our notice was 40 days in advance (if you assume 30 days standard, thats a tricky little gotcha). Luckily for us I sent these 2 months in advance.

So disconnect for site 1 was supposed to be December 12th. On Friday (January 6th), they sent an outage notification for site 1 to our billing department, despite me having set up three points of contact with their technical department, with a specific order to be called in. Me, The Dev, and The Boss. I had confirmed these contacts with them twice. (I've learned you see.) But not a single email was sent to anyone technical.

Today, (January 9th), they called two of our technical contacts, at the same time, without mentioning that they were talking to both of us. I mean, if it had been a real outage and not just them opening trouble tickets on their own disconnects, the three day turn-around might have really pissed me off.

Also, considering we paid for BGP service and dual links at dual sites, the steps they list in their outage emails are excellent.

To ensure the proper testing is done on your circuit we ask that you email or call our customer care team with the following information:

1. Local Contact Name, Number and access hours to the location
2. Have you verified the power and equipment? (Lights on router, rebooted router and check all connections on the router)

Right. Because if you are a BGP peer on the internet and you have a problem, power cycle that baby like shes a $20 D-link at grandma's. And if you think this is just the form email, rebooting the router has been their "network engineer's" suggestion three times out of four.

When I wonder how we ended up buying anything from them in the first place, I just remember the request IT received last week from a certain high level executive. He needed a 30ft network cable for his television. When we offered Cat5e, we were informed he needed Cat7 as he has a 3D TV.

...

Out.

When good NICs go bad.

2012-01-08T19:03:00.001-05:00

Having some trouble with switch local latency on our network, I came across this packet:

00:00:00:00:00:00 > 00:00:00:00:00:00 null information, send seq 0, rcv seq 0, flags [command], length 46

This was only spewing maybe once a second, but not knowing how random hardware deals with what amounts to a fuzzing attack, I came in on a Sunday to track it down.

After isolating it first to a switch and then to a pair of ports, I traced it back to a pair of Dell Optiplex 380s. The strange thing is this only happens if the machine is in sleep mode. Both machines use S3 sleep without any WOL or Dell image server configuration. In fact both machines have a specific bios configuration that warns 'The onboard NIC will be shut off to save power.' Then why does the sleeping machine maintain link lights?

Why only two machines out of an office of 40 some? And why are the two machines right across from each other? I guess it could be a power issue. Changing out network cables to known good ones using a different path did not change anything. Other devices (phones) using the same ethernet cables don't show this problem.

For now I make sure neither machine can sleep so I can.

There are more things in heaven and earth.

2012-01-06T17:02:00.002-05:00

I've often tried to explain that the choice to make a highly available system should not be taken lightly, as your budget will always be limited while the failure modes are infinite. Example: if Level3 only would have invested in that bulletproof fiber, they could have avoided this downtime in '07. Systems are full of strangeness that you may never catch.

Today I'm working on building a new routing/switching core. These have been my workhorses in my environment for a while; our data size is small and firewall count is high. At 600$ a piece, these gigabit 1ghz embedded boxes run Vyatta and pf well. I'm hooking all 4 gigabit ports to the same switch, trunking them with LACP and then using static VLANs for segmentation. This design has served me well in the past as the physical layer (cables) is 90% abstracted, making remote changes a breeze.

With everything cabled, and all three routers powered off I went to configure my switch. On power on, instant switching loop. So after standing there for a while scratching my head at activity lights on ports connected to unplugged equipment, I disconnected everything, checked the cables, reset the switch and eventually found what I was looking for. When these routers are unpowered, ports 3 and 4 become electrically connected in a correct enough manner for data to flow. No other ports become connected like this.

Crossover Wat?

I won't worry about it much, LACP will fix this issue. But what if I had put this into production and a router had lost power? I don't run STP on these core switches so the routing loop would have taken out the whole network. Let this just reinforce my point, you can't plan for everything.

Touchpad Scrolling Mini 311 (Alps Touchpad) FIX/SOLVED

2011-09-09T21:32:00.001-04:00

To Get Scrolling working, copypasta this into a terminal:

echo "options psmouse proto=imps"|sudo tee -a /etc/modprobe.d/psmouse.conf; sudo modprobe -r psmouse; sudo modprobe psmouse

This is mostly for my own benefit so I remember. I have a HP mini311 laptop, and on default install of Ubuntu scrolling doesn't work. This doesn't fix all the fancy features, but it does make scrolling work. This is a long standing kernel/xorg bug where the touchpad is detected both as a plain PS/2 mouse and as a touchpad, but Xorg listens to the first one. This tells the kernel to treat it as a PS/2 mouse with a scroll wheel.

Simple nuff. Should seriously be fixed.

Bandwidth.com Security problem

2011-05-25T18:27:00.002-04:00

So about a year ago I had to reset a connection to a manged router we bought from Bandwidth.com. I needed a password to access the console, and the tech gave me it as username: "bandwidth" password: "b@ndw1d+th" . I thought it interesting at the time, and wrote it off as the default password on new routers. The tech would have of course changed it.

Next day I had a few minutes, and curiosity got the better of me. So I tried SSHing into that router, and it didn't work. I was happy. Just for shiggles, thought to try telnet. "Who uses telnet on the internet anymore?" I thought. But no, telnet was on! And worse, that default user/pw worked. More suspicious, I decided to try another bandwidth router we have in another colo. It had been installed more than a year previous, and surprise: it worked there too.

I of course changed both passwords, informed my boss, and had a nice talk with one of their techs who explained that is the way they send them out. I was more than slightly pissed, but it had been fixed, so I dropped it. Skip forward to now, I have to work on those lines again, apparently a BGP session went down. In between waiting for some info, I told the tech I was working with about it, and asked "You guys don't seriously keep the same username and password on all your routers right?" The reply was "I can't say anything about that." Not a denial, not a wtf, not even a blatant lie.

Googling around, I found this. http://uc500.com/en/security-issue-bandwidthcom Apparently this isn't the first time Bandwidth.com has been caught doing something stupid. There is a comment in there saying to call Sean Rivers, a director for their voice group. I wanted to ask someone in their management chain to see if they had fixed this issue, if it would be safe for me to write a blog post about it. I wouldn't want for someone to start breaking into all their routers and shutting down links, as my SysAdmin counterparts would have to deal with it, and thats just not neighborly. Nor does it follow responsible disclosure.

So I called Mr. Rivers. He informed me they had fixed this on all their Edgemarc voice routers, and I would have to contact Jeff Witles? I called in, left a voicemail, and got a call back from a Bob Willingburg, one of Jeff's subordinates. After explaining what I had found, that this seemed to be a bad security issue, Mr. Willingburg explained that setting a default password was policy, and that their team always would work with customers to change it. And that was that.

So heres this. I know no team worked with me to change the password. I found it accidentally. As I was told, these were managed devices. I don't know about everyone else, but "managed device" means I don't have a login to it. I don't know if that password is default on all their routers, or only the Nortel SR1002 I found it on. I'm assuming they have larger Cisco/Adtran devices for other customers. If you are a Bandwidth.com customer, try it. Hopefully this will let you catch it before the rest of the world does.

G1 Headset Adapter Pinout

2010-01-09T22:23:00.010-05:00

Update, years later, I find this on Amazon. http://www.amazon.com/gp/product/B002SK66OY/ref=oh_o06_s00_i00_details

Don't trust this too far, I just got this working hooking the microphone directly to the board in place of the other one. I lost a solder joint so now I'm going to wire it directly here in a bit. Just wanted to upload what me and my angry multimeter found with this adapter. I wanted to hook up my Logitech gaming headset (which has 2 3.5mm jacks, one for headphones one for mic) to my phone, and it pissed me off they didn't use a 4 pin jack so I could make a simple adapter.

This pinout would be if you took the little adapter apart and were looking directly at the board.

Also:
Left and Right Audio are Red and Blue.
Mic is White wrapped in its Gold Ground.
Gound is Gold.
Activation is Red/Green.
Button Press is Green (Remember 100ohm resistor).

Finished Product among the mess of my desk.

Working with VDI files

2009-12-10T17:52:00.002-05:00

Sun's default harddrive image is just a little padding onto a raw disk image. I needed direct access to disks and partition within one of these images, which takes a little finagling. As far as I know, this only works when the device is FIXED and no snapshots have been made.

This blog post gives us the raw details of how to access a VDI file. Working off that, some clever (perl) fellow made a script to mount the a directory out of it. It can be found on the Arch User Repository here.

To do this manually, download the vdiinfo.c file from here.

Run "gcc -o vdiinfo vdiinfo.c" to compile it.

Run the tool against your VDI file. "./vdiinfo SomeImage.vdi"

Mount the harddrive to a loopback device using the "Offset Data" value returned by vdiinfo.
"losetup /dev/loop0 SomeImage.vdi -o OffsetData"

Now to find the partitions, you can use fdisk.
"fdisk -l -u /dev/loop0" will return all info raw on the device just like a normal hdd.

Say we want to access the partitions directly for some tools, like partimage. To do that, we can mount it again. This one is a bit more interesting. You use the start of the partition returned by fdisk, multiplied by the sector size returned by vdiitool as the offset. $(()) is bash mathematical expansion.
"losetup /dev/loop1 /dev/loop0 -o $((sectorsize*partitionoffset))"

And viola, now you have a partition in a vdi as a linux device file.

Fuck Gen-u-swine.

2009-11-11T11:22:00.005-05:00

An APC battery unit slides into a rack, with 4 to a shelf. This is the unit for our Symmetra PX UPS. These use 8 standard off the shelf 12v 7Ah battery. APC isn't the manufacturer BUT they want to charge 1700$ for a brand new unit, completely ignoring that its 8 phillips screws and 8 new batteries to have it good as new. Even at the outrageous price of say 80$/battery (Single price, I've yet to get the quote for 448 of them) thats only 640$ per unit.

And yet even though APC is simply a reseller for anothers battery at a 400% markup, they still try to convince you through FUD that other resellers are evil and could be dangerous.

Don't believe the FUD.

Update: Got the quote. 13$ per battery. 5k for the whole UPS.
Update2: Installed the batteries. Works wonderfully, passes the calibration test as expected.

SysAdminBot Powers Up: Destroys EVERYTHING

2009-09-24T22:59:00.007-04:00

I just started with a company 2 weeks ago, and the number of black boxes is inversely proportional to the amount of documentation. Its looking to be a nightmare.

Machines that I didn't configure are hard enough to deal with, but these were configured incorrectly, compiled from source, out of date programs and SQL systems, strange options, hacks, with the odd perl daemon tieing it together. Package Manager? Lol. Not to mention an entire high density APC InfrastruXure vault of 20 racks and a comms closet with the remnants of no less than 6 different external connections. I'm still working on counting them.The patch panels are a mess.

We have wires over wires, often disconnected 5 years ago at both ends (I've found tags) and left in the channels. There have been at least four previous admins with the most recent being a web developer. I use the term SysAdmin as loosely as a hooker at 4am. The wiring is nothing compared to the machines themselves. The company employing me didn't build this but rather bought it from a hosting company. The hosting company rigged it together, realized it was too small, then left and took everything useful with the rest consigned to us in the sale. There were over 50 1U and 2U servers running in this "datacenter" continuously since the purchase a year ago. Luckily a few months before I started someone pulled their main routers internet connection, but up until then they had been working ... doing... something. There still is some obscure flight simulation datasets on a Windows 2003 server that I'm supposed to be on the lookout for.

There are multiple phone systems including our asterisk box, and Atavan system, and an ancient Avaya system at a remote office. And we are putting in two more. Some of the infrastructure is lying out on desks, in closets, with switches and routers and an out-of-date Linksys running OpenWRT with a hacked GUI tying it together. The WRT randomly looses routes and doesn't apply iptables settings, and now has a bash script croned holding it together once a minute. There are three different CA's that we buy from for SSL certs. One office has enormous latency and another has a comms closet that when a wire is nudged will bring down the specialized financial frame relay. Which is slightly amusing, consider this link has redundant routers with separate T1s and separate backup ISDN lines. None of which apparently work.

And what to do about it? I'm ripping out everything. I'm learning anything I need to know to keep things working just long enough to replace them. Asterisk boxes are coming in; I'm trying to convince the CTO to slow the expansion and I have the backing to do anything that needs done to make things right. I've already disassembled the datacenter, finding four machines colocated from our ISP that they knew nothing about. I now have a nest of wires up to my knee, and machines everywhere.

500$ worth of wiring and cable ties and management are on their way. Crimps, tools, labels are on their way. When encountering a black box, there is a method of operation. Find, Isolate, Duplicate, then in a few hours in the middle of the night, DESTROY. I shall have my system. And it shall be done right.

I am like a robot consuming systems. I map them, cut their cruft out like a cancer around a vital body part, and then strengthen it by throwing new machines at it. A tip: The most useful thing I've found is a battery operated Black and Decker drill, which I bought myself for 20$ at the home depot. I can strip a machine in under 3 minutes. The Robot Admin keeps on rockin.

How to Update drivers for Ghost Boot Images

2009-05-13T16:26:00.006-04:00

Intro
So this previously was a post on TechSpot. A year ago, I had no publishing setup, no blog, no twitter, nothing. (I was happy too.) So when I finally fixed a problem with Ghost at work, and wanted to share it with the community, I googled "IT Technical Forum," and created a thread on the first link that popped up. You can find that post here.

People seem to be having problems with that, so I thought to repost with a bit more explainations. "If you teach a man to fish" as it were.

To mention; we are mainly a Dell/Windows shop, with Mac and Linux scattered around. Change as is applicable.

Symantec Ghost: Dammit DOS
Ghost has two executables. One 32bit that works in Windows proper, and one 16bit executable that is built for DOS, be it Free-DOS or MS-DOS. These used to be floppy images, but most people now will use them on USB flash drives. (It can also make ISOs to be burnt to CD. I'm sorry if you're still working on machines that can't boot from USB.) The "Ghost Boot Wizard" in our version of Ghost Enterprise 10 can automatically create these flash drives. These can connect to a Ghost server session, but to do that Ghost has to configure DOS with the proper network driver, or it simply won't work. Nothing fancy as Linux/udev here.

When using the Ghost Boot Wizard, we select "Network Boot Package" and then the driver for the card we want. Select some options for Ghost, etc etc, and viola it flashes a USB key for us. Walk to a brand new Dell Optiplex 755, Optiplex 960, or Latitude E6500, and it returns "Network Card Not found" because these machines use chipsets not in existence at the release of Symantec 10. And of course proprietary software gets no rolling updates, as that wouldn't make the next release as profitable.
Be it DOS or Symantec's problem, that just won't do.

Express HOWTO

Go to

C:\Documents and Settings\All Users\Application Data\Symantec\Ghost\Template

In the folder for the card, update the .dos file to the newest, and add the pci-id of the card to the mcassist.cfg file.

I've uploaded my copies of the Broadcom BCM57xx and Intel Pro 1000 ghost template folders to http://drop.io/ghost_update_files as well as the Unknown Devices Utility. I know these to work on the follwing models: e4300, e6400, e6500, o755, o760, o960, d630, d620, d610, d600. You can always just drag the driver and mcassist.cfg files into your folders and it will work.

Full HOWTO

The Network Drivers for Ghost are stored in

C:\Documents and Settings\All Users\Application Data\Symantec\Ghost\Template

Go ahead, open this up. Its pretty. The two most common network cards from Dell are built by Broadcom and Intel, namely the Broadcom BCM57xx and the Intel e1000 based cards. Each of these drivers work with a multitude of NICs, the problem is making sure the Ghost autodetect script knows which driver to load.

Go to both

C:\Documents and Settings\All Users\Application Data\Symantec\Ghost\Template\Broadcom BCM57xx

and

C:\Documents and Settings\All Users\Application Data\Symantec\Ghost\Template\Intel PRO 1000

Note: These directories of course can be different, or not exist at all. If they don't, download the latest Broadcom and Intel drivers, and add them using the Ghost Boot Wizard as described by Symantec here. You can also get drivers from the specific machines at support.dell.com.

Note²: The files in these folders may be hidden. Show hidden folders as explained by Microsoft here.

Note³: If you already have these drivers, make sure your version of b57.dos and e1000.dos in these folders support whatever card you're using. Older drivers, though named the same, will not include support for chipsets created after their time.

Right then. The key here is the mcassist.cfg file. This crutch contains a list of PCI-IDs that work with the given adapter. Add another PCI-ID to this file, and Ghost will know to load that specific driver if its scan returns that PCI-ID.

To add this information, you need to know what the NIC's device ID is, as well as its chipset model. You could always run lspci in Linux, but this isn't Linux. Since most of what we support is the crippled backwater of an OS, you can use the Unknown Devices utility in Windows. It will look something like this.

With this knowledge, we add an entry that looks like this.

PCI-TAG = 14E4 16F7 : Broadcom BCM5750A1$

Recreate the USB flash drive, and this time it will work.
Conclusion
I could never get Ghost's auto detect to work right when both the Broadcom and Intel drivers were on the same flash drive. I have fixed that now, and in the process made UBER FLASH DRIVES with the Ultimate Boot CD project. I just finished them and distributed them to my department today. Just think, every single boot image on one flash drive. That in a future blog.

I found this after the fact, but Symantec has a howto on this.