waterskiing!

So this morning we went to Ulsoor for rowing. Sindya and I did ran around the lake and were just stretching and waiting in line for a single-scull. Today was the day we were meant to practice balancing in a scull while tied to the pier.

Just then, they brought out some waterskis and there was an army guy trying to learn how to waterski. He wasn't able to get out of the water, and I made some comment about his skis not being positioned correctly. One of the coaches overheard so he turned to me and asked if I knew how to ski. Then he was like "will you show us?" It was issued as sort of a challenge. So I said that I could definitely give it a try.

Well eventually the boat came in and I got into the water. The only thought going through my head was "shit. now a bunch of army guys are going to get a laugh out of me falling" Luckily, I was able to get up on the first try and even cross the wake a few times.

After two rounds of the lake I came back to shore and watched Sindya's balancing act in the scull (she had already gotten in). There wasn't time for me to try today, but that's okay. I haven't waterskied in oh about... fifteen years. So I've spent the rest of the morning smiling from ear to ear.

PS: I should mention that Ulsoor was no cleaner today than it was yesterday. So if I die because of sewage ingestion, then we all know why. :) (actually it wasn't that bad)

gluttony

Moral: Gluttons shouldn't go to an all-you-can-eat place when starving.

Story: After rowing this morning and attempting to improve my running time (no luck: still did 16:54 around the lake), I rushed to the office and forgot to pick up breakfast either at home or near the office. By 11am I was starved and light headed.

Every Wednesday they have an all-you-can-eat salads-and-cold-cuts buffet at a german cafe (Cafe Schlomegger) above the Max Meuler center in Bangalore. I called and made a reservation for 12:30 (when they open). I ate about four full plates of meat and salads, after which Sindya (who didn't eat quite as much as me) proceeded to order dessert. Of course, I had to have some of that too.

After a 90 minute lunch, I could barely walk the four flights of stairs back down to the road. An hour later I'm at work feeling unbearably sick. ugh

On a side note, Moohaha writes about bad drivers in the US. I'm amused

running

The rowing coaches want us to be sufficiently warmed up before starting to row, so I've been running around Ulsoor Lake every morning for most days the past two weeks (the long weekend excluded).

I was told that it's 3.6 km around the lake.

I timed myself the past three days and the times were 18:34, 17:40, 16:54 respectively. Pretty rapid improvement once I started timing myself, but I know I'm getting closer to my current limits.

As soon as I get under 15 minutes I'll be close to a six-and-a-half-minute-mile, which is my current goal. I'm told that the Army dudes run it in under 15.

Given that I do an hour of rowing after that, I'll probably just stick at that, until it becomes painlfully easy for me (wishfull thinking)

Kerala

Will update this entry later; need to head to bed. But here's a link to photos from a weekend in Allepy, Kerala.

(If you're subscribed to the feedburner version of my feed, you should already see the pictures)

So last Thursday night, Sindya and I took off for Kerala. We got in a bus at 4:30pm, and reached Allepy at 5:15am on Friday. A car from the resort was waiting to pick us up. We got to the resort minutes before sunrise, and were able to take some great snaps. The resort itself is very rustic. No TV in the rooms, all wood construction, outdoor openair bathrooms, etc. It was so beautiful there that we just stayed up and watched the sunrise and hung out in a hammock until breakfast time.

For breakfast, we had Appams and Stew - a traditional Kerala breakfast. The stew is basically vegetables boiled in coconut milk - it's delicious. After eating about 4 appams and exhausting the stew that we got, I asked the waiter to bring me a bowl of coconut milk and a jar of sugar, and proceded to eat six more appams doused in coconut milk and sprinkled with sugar. That about set the tone for the rest of the trip.

At around noon we took all our luggage and checked onto a two bedroom houseboat (there was no one in the other room so we had it to ourselves). There were three people on the boat with us; a fulltime cook, a dude who steered the boat, and a dude to run the motor. :) The cook made some amazing lunch (shredded coconut in some kind of gourd, some daal, and a yummy kerala fish curry) and dinner.

There was a mattress on the front of the boat where you could just lie under the sun and watch the world go by. That's where we were most of the time. We stopped in the middle of a lake for a while and I went for a swim though was a little scared because the dude told us there were water snakes (non poisonous) in some parts. At around 6, we docked somewhere and went for a walk, had dinner, and went to bed.

We got back to the resort by about 10am the next morning (breakfast was just toast along with an amazing dessert of caramalized unripe banana - why is it that we don't have dessert after breakfast all the time?). At noon we got on another boat to go watch the annual snakeboat races.

Each snakeboat accomodates about 120 people (usually representing one village). There are several heats with 4 boats each; it looked like they rowed about 2+ miles but I'm not sure. It was great fun to watch the first 3-4 heats but it got old quick.

After finally making it back to the resort at 6pm, we hogged some more on the buffet dinner (appams and coconut chicken curry, followed by many more appams eaten with - you guessed it - coconut milk and sugar) and hit the sack.

Sunday morning was spent just chilling in preparation for a 4pm bus back to Bangalore.

Overall it was an amazing trip. Did I mention the coconut milk?

innovation week

A few months ago, I had a really kickass idea for a relatively simple web app that, I think, could make web surfing much better (yes, another one of those ideas). I worked on a proof of concept but, being close to a big release at work, I couldn't find the time to concentrate on work as well as a prototype.

With support from Management, I was able to spend all of last week (minus a few hours each day when I had to attend meetings) working from home on my prototype. It was great fun to stay up late with pizza, coke, and some good music and hack away; I haven't had that much fun coding in... well... a long long time.

On Sunday night, I slept at 10pm (because I was going to start rowing classes at 6am on Monday) but ended up waking up at 2am with a head full of code. The last time that happened to me was probably during OS class way back in 2001. I ended up coding from 2am - 5am, going to the rowing class at 5:30am (we ended up not starting until today so just went running), playing tennis from 8-9am, and then being at work all day. Just when I thought I was ready to pass out at home in the evening from exhaustion, I just had to bust out a few more features that were spinning around in my head.

The app is not ready to go live yet, but it's at the point where I was able to demo it off to a few people at work today (sans any browser crashes or error messages). It's the first time we've done an 'Innovation Week' at Amazon Bangalore. Working on something small and innovative gave me a badly needed recharge; back to 'real' work now.

I'm totally convinced that RoR is the way to go for frontend development. It's funny; Ruby on Rails made the webapp/db coding so trivial/nonexistent that 90% of my time was spent writing javascript. I've mostly been doing backend, server framework-type coding at work so I'm not exactly an authority on frontends. But the little jsp that I've been exposed to tells me that RoR is the way to go.

rowing

As of this morning, Sindya and I are signed up to row for an hour every morning on Ulsoor lake, from 6:45 - 7:45. It's *amazing*. We're spending the first week rowing on a fixed boat (it's tied to the dock) to learn the technique.

The location (Trishna Yacht club) is Army property but they allow civilians to use it, through a company called Catapult Connect. The coaches are all super friendly Army guys who've competed nationally or internationally. Six days a week for 23 more days, we'll be waking up at six, taking a quick run around Ulsoor Lake, and then rowing for an hour.

rails on fcgi

I've been working on prototyping a small rails app at work this week. On Friday I decided that I had enough functionality to be able to do a basic demo, so I decided to focus my attention on getting the demo up and hosted before getting back to fleshing out the functionality.

Since then I've been in config hell. Things were running really slow on my box because I was using CGI. Like an idiot I decided to switch to using lighttpd->fcgi->rails instead of apache->cgi->rails, but do it all in one go. I can't believe I still don't know better than to try doing multiple switches like that in one go.

Anyways, I came to my senses this morning and followed the following excellent instructions on getting fcgi set up on my local box. After (almost blindly) copy-pasting the commands, my app is up and running a lot faster with fcgi support.

The next step is to make the apache->lighttpd switch.

out of this world.

this is nuts. check out the moon.

And, for shitsandgiggles, make sure you zoom in all the way.

hat tip, google blog.

no way!

a multibutton mouse from apple!

One wonders why exactly it took them over a decade to realize that it's useful to have more than one button. Though I agree that more buttons is not always better; I have almost never used the 'forward' and 'back' thumb-buttons on my Microsoft mouse.

amazon 10 yr celebration

SO amazon.com had quite a big celebration for its 10 yr anniversary; a concert featuring Norah Jones and Bob Dylan.

We had a company picnic/10 yr celebration in India as well. We went to the Kings Sanctuary resort in Nagarhole. We left on Friday morning with two big buses and a small tempo, and arrived there in time for a late lunch. The rooms there were pretty amazing, and Sindya and I were lucky (with a little help from Rekha as well) to have a cottage to ourselves (most other people slept 4 to a cottage - two in the bedroom and two in the living room).

We went out for a safari that afternoon. We didn't see any tigers; we just missed seeing one by about 10 minutes. Ironically, people travelling on a loud, interstate bus saw a tiger whereas we (travelling in jeeps, specifically there to see the wildlife) didn't. We got to see lots of spotted deer, some Sambar deer, a wild elephant, a Gaur (I think it's the same as a bison), two types of monkey (langur, and one other) and a bunch of birds whose names I forget. We got pretty cold and wet, it rained the whole weekend. After the safari, we had dinner and drinks. A couple of us went for a midnight dip in the pool.

Vikas and I had brought some climbing gear so at 7:30am on Saturday twelve of us got into the tempo and went to a nearby rock for some clibming and rapelling. Vikas and I set up rapell with the help of the guide (he actually is an entertainer at the resort who'd just learnt to climb by watching people set stuff up there). Apart from the two of us and Pooja, no one else had rapelled before so they needed a lot of convincing to lower themselves off the edge of a cliff. Pooja required the most ;) We didn't have time to climb, but I think everyone had a great time. Shivam and Harman came away with a few bruises since they were the first two (after Vikas) to try rapelling down so didn't have a good idea of how it was done. Sorry guys!

We got back to the resort, played in the super-cold swimming pool for about 45 minutes, had lunch, and then got back on the bus to head back. It was a good break.

Now I have to plan a real honeymoon before the wife kicks my butt!

catching up on reading

learning from mistakes:

For anyone than never discovers a deeper self-identity, based not on lack of mistakes but on courage, compassionate intelligence, commitment and creativity, life is a scary place made safe only by never getting into trouble, never breaking rules and never taking the risks that their hearts tell them they need to take.

I think I often fall into a rut where I get scared to take risks because of the above. It's good to read shite like that every now and then to jarr me out of my apathy.

The curious incident

There were two chapters I absolutely loved in this book:
On death (chapter 61)
and on life (chapter 199)

(you should be able to read those excerpts online thanks to amazon)

Random

Posting has been light because I've been

• sick
• trying to figure out the intricacies of married life :)

So here's some random stuff:

1. Since I started working, I never really made a conscious effort to save; I just made sure I didn't splurge. The result was that I was able to put away about 40% of each month's take-home. I've been following the same strategy in India - making sure not to splurge, but definitely not being super-conscious of my spending. The result is that I'm not able to save much money at all. That bothers me.
   
2. I have a fair amount of indian classical music on my ipod. I manage to go through my other music collection using the 5-star rating system, along with dynamic playlists. I hadn't found a good way of going through my classical collection though. Using information from this page I've managed to build up 4 playlists (morning, afternoon, evening, night) so I can listen to the 'appropriate' pieces. However, I'm now longing for even finer-grained control of my collection. I wish I had a SQL front-end to the iTunes db.
   
3. One of my friends (Vikas) has an awesome cook, who's agreed to come cook at our place once a week. She makes awesome food. She's given me a list of stuff I need to buy though :) belan (rolling pin), some plastic containers, a bunch of specialized steel kitchenware (container for flour, plate for kneading dough, etc). To add to my already-over-equipped kitchen.
   

Oh and, just so it doesn't get lost in the jumble above, I have to mention separately: you MUST check out prince of ayodhya - a contemporary rewriting of the Indian epic Ramayana. It's the first in a 6-book series, four of which have already been published in India. I suffered the wrath of my wife for burying my head in these books the past few weeks. They're amazing. The story itself is so gripping, and Mr. Banker just makes it even more irresistable by retelling it in a vernacular that's familiar and easy to follow.

I'm at home house-sitting while the cook and maid do their thing. Sindya is out running a program for Dream-a-dream that she's planned, called Monsoon Madness. It's a day long talent-show/competition for underpriviledged kids. They'll be painting, dancing, acting, taking quizzes, and making collages. As soon as the maid gets done, I'll be back there shooting video and hanging out with the kids. Until then, I'm a domesticated husband (damn that sounds scary).

back from my wedding

To check out pictures from my wedding, subscribe to this feed.

If you took pictures at my wedding and want them as part of this slideshow, upload them to flickr and tag them with 'sindyaandhemant' as well as 'wedding'.

More updates later.
Update: flickr's rss feed seems broken. the atom one works okay (link updated above), however it only shows the last 10 photos. Since I'm uploading a bunch of them, you'll miss out on some photos. You may just want to check out the slideshow instead.

lightpost

posting will be light for the next few weeks.
In a few hours I fly to Delhi.
Tomorrow morning I fly to Leh, Ladakh for a 5 day trek in the Indian Himalayas.
After that I head back to Delhi and on to Pune for about 10 days during which time I will relinquish my bachelorhood.
I may be online and checking mail from Pune but probably not posting. Will be back to the grind sometime in July.

prediction

I wrote parts of amazon's remote-service-invokation framework a few years ago, after which a few of us worked on collecting detailed metrics from the framework so as to guage the health of the system and try to predict business-impacting problems before they actually, well, impacted business.

I'm currently reading blink by Malcolm Gladwell and it's a facinating book. No time for a full review right now. But here's a quote that caught my eye (he's talking about an algorithm for deciding the seve rity of heart-attack-like symptoms in ER patients, and has just listed several high-heart-attack-risk lifestyle factors):

... It certainly seems like he ought to be admitted to the coronary care unit right away. But the algorithm says he shouldn't be.
<snip>
What Goldman's algorithm indicates, though, is that the role of those other factors is so small in determining what is happening to the man right now that an accurate diagnosis can be made without them. In fact <snip> that information is more than useless. It's harmful. It confuses the issues. What screws up doctors when they are trying to predict heart attacks is that they take too much information into account.

(The book, by the way, attempts to explain intuition and how it is that we can get such strong (and often correct) intuitions without being able to understand exactly why. It also attempts to analyse the cases in which our intuition is terribly wrong. See also this entry by Trevor for more about intuition.)

This is cool because our hunch over the past few years has been that it will only take a few metrics to actually predict a given failure scenario, but deciding which ones to pick is the hard thing. So the kinds of systems we are trying to build end up being quite similar to what (I just found out) humans are doing. We're constantly taking hundreds or thousands of input variables (subtle changes in a persons face or 'body language', things seen in the periphery of our vision, etc.) and doing some realtime statistical analysis on them. Except our consciousness is never burdened with any of that. Our subconscious builds and refines these elaborate statistical models over time. Then, it can bubble up signals (in the form of intuition) to our conscious mind with very limited information because it has already made models about which variables are important enough to matter.

How does this apply to metrics and monitoring? It's infeasible and foolhardy to track the state of every possible instrumentable variable in your system in realtime and use that to drive failure detection and root cause analysis. But

1. you may be able to design a system that can collect lots of metrics and analyse them in an 'offline' manner without impacting your system.
   
2. the output of (1), a list of 'important' metrics, is fed into an alarming/monitoring system
   
3. whenever an alarm is diagnosed (or confirmed) the result of that is fed back into (1) to correct or reinforce the prediction.

If failure detection is like the pit in your stomach or lump in your throat, and root cause analysis is like the logical reasoning that we sometimes go through when making decisions, then maybe we have to accept that failure detection is a much faster process than root cause analysis. Our group has always looked at those as two different processes, but never acknowledged that they may require different amounts of information.

On one level, that looks hopeless; "what good is it to know that something is wrong if you don't know what it is?" But we do that all the time. A lot of us learn to trust our instincts (don't walk down that alley) even if we can't tell exactly what's wrong (it's well lit, there are people around, but it just feels shady).

How could that help in managing distributed systems? The only example I can think of right now is: if a host 'feels like it's unhealthy' it could just take itself out of a load balancer without knowing what was wrong.

It does tell me is that it may be worth completely separating the process of detection and root-cause analysis. So that the feedback in (3) above is not "the root cause of this disturbance was xyz", nor is the list of 'important' metrics in (2) a list for each possibel root cause. (i.e. you don't output something that says that metrics A and B are important for predicting a disk crash, but metrics D and F predict a web server failure and metrics C and E predict that your application is deadlocked). That's how antivirus software is modeled. It builds up fingerprints of different viruses and tries to match the fingerprint. It does both detection and root-cause-analysis in a single step. (OK, maybe modern antivirus software does more than that, but stick with me for a moment).

Instead, maybe the right but counterintuitive (no pun intended) thing to do here is to only store whether or not "Bad" things happened, and store the set of metrics which are good predictors of "Bad"ness. You'd probably need more than a binary notion of Badness. This doesn't get us closer to solving problems, but maybe it can help reduce downtime in the first place, because we've got a very good early-warning system.

It'll be interesting to see if any more insights come out of watching a large system running (the group I've been working with in Bangalore is getting closer to releasing an internal, scaled-down version of what will eventually be a large self-healing distributed system). Since I've been in development mode for the past few months (vs. supporting a live system), I feel a little unqualified to rant too much about this stuff. :)

d.s.

Someone left comments (twice!) asking me to follow up on my earlier idea of starting a distributed systems blog. Thanks! But I realized that, although I do a lot of work in the area, I don't know enough to do justice to a blog - I still have many miles to walk.

So every time I run across something interesting, I'll be sure to post on it. But until then, I'll be sticking to reading and learning and growing.

FC reading.

catching up again on some FC reading.
About Despair, Inc.

The point is that most people should work to make money. They shouldn't expect a company to make them happy. A company can be friendly and good, but it can't really make you happy. At the same time, it shouldn't insult you. It shouldn't say, 'We're a family and have values,' and then act like Enron."
<snip>
Jamie Malanowski is the features editor at Playboy. He's happy in his work.

I love the last line about the author of the article! :)

An awesome post about being a bouncer:

The bouncer ethos, in point of fact, stands in diametric opposition to that of any other position in the service industry. Simply put, if you, as a bouncer, stand there and take crap from the customers, you won't be employed for very long, because everyone on the staff will consider you a pussy, and they won't want you around. Therefore, when people -- as they invariably will -- act like assholes, I'm getting paid to fulfill the one, singular fantasy harbored by everyone who has ever served a drink or waited on a table:

I can do it back.

On cubicles sucking:

The solution, Tompkin says, is to customize space to various types of work. Give those who need uninterrupted time a quiet place to work and those who need to collaborate a more social space. That may mean a glass-walled office for heads-down work, and a variety of gathering places for group work. "As the workforce becomes more mobile," Tompkin says, "the office will be the main tool companies use to build a shared culture."

I totally agree. I wish we'd have a healthy mix between offices and cubes. Or even moderately sized rooms with cubes in them, instead of a huge room full of cubes. That way I can only potentially get distracted by 4-6 other people instead of 70.

Sounds like GE Durham has taken a page out of Gore's (makers of GoreTex) book (Reference to gore from Malcom Gladwell's tipping point. Lots of great stuff here:

GE/Durham has more than 170 employees but just one boss: the plant manager. Everyone in the place reports to her. Which means that on a day-to-day basis, the people who work here have no boss. They essentially run themselves.
<snip>
So how can something so complicated, so demanding, so fraught with risk, be trusted to people who answer only to themselves? Trust is a funny thing. It is the mystery -- and the genius -- of what goes on at GE/Durham.
<snip>
"The interview, now that was one heck of an experience," he says. "It lasted eight hours. I talked to five different people. I participated in three group activities with other job candidates. I even had to do a presentation: I had 15 minutes to prepare a 5-minute presentation."
<snip>
At GE/Durham, candidates are rated in 11 areas. "Only one of those involves technical competence or experience," says Keith McKee, 27, a tech-3 on Team Raven. "You have to be above the bar in all 11 of the areas: helping skills, team skills, communication skills, diversity, flexibility, coaching ability, work ethic, and so forth. Even if just one thing out of the 11 knocks you down, you don't come to work here."

Some of the stuf here reminds me of what we learnt in the storytelling training I attended recently.

cliched conversations

As a foreigner in the US, I was always struck by the very predictable 'conversations' that people had:
- so how was your labor day?
- what're you doing this weekend?
- how was the weekend?
- got plans for july 4th?
- going anywhere for memorial day?

I put the word conversations in quotes because it seemed a lot of times that people weren't even interested in the answers, they were just asked because of a strange notion of politeness.

I'm sure it's a universal phenomenon and not at all restricted to the US. But I can't come up with similar examples in Bangalore.

Except for one... for the past three weeks, people have constantly been asking me how preparations are going for the wedding and if I'm all set. Worse still, the same person will ask me the same question two days in a row, as if my 'preparations' change on a minute-to-minute basis.

Just like the conversations above, a lot of times it feels like people are just asking me to be polite. Especially given that I've been telling them that my mom is doing all the preparations. (An upside of having the wedding in India is that my parents and family are doing a lot of the preparations).

Actually I'm kind of nervous that I haven't had to do much. You know that dream where you show up in school and everyone's laughing and you look down to realize you forgot to wear your pants? I have that same feeling. Like I'm going to show up in Pune and realize that I forgot to do something really basic. :)

Anyways, I'm leaving for Delhi this Monday, after which I'll spend a week trekking in gorgeous Ladakh. From there I fly straight to Pune to tie the proverbial knot.

phishing

Here's an interesting way to beat phishers and their scams:

If you get phishing e-mail, go the web sites and enter false data. Make up everything -- name, sign-on name, password, credit card numbers, everything. Instead of one million messages yielding 100 good replies, now the phisher will have one million messages yielding 100,000 replies of which 100 are good, but WHICH 100?

This technique kills phishing two ways. It certainly increases the phishing labor requirement by about 10,000X. But even more importantly, if banks and e-commerce sites limit the number of failed sign-on attempts from a single IP address to, say, 10 per day, theft as an outcome of phishing becomes close to impossible.