I always tell our guys that I don't care what they do, where they do it or how busy they are - I only care about the result being there and any information I might need to be given before it is too late. So I'm actually happy when people aren't overworked and double happy because I get volontary engagement if something needs to be done quickly. Live is too short to waste it by always only thinking about money.
Well, that's nice to hear. The quote I stated though is precisely why I didn't fully get into I.T. Even the I.T. teacher I was doing voluntary work for in my high school when I was younger, took the 'dumb slave driver' approach, instead of being reasonable and logical.
I would suggest that you and a few others are not the norm, but the exception to the rule.
That needs to change.
In my potentially less than humble opinion, I.T. needs to get a certain level of authority over those who we work for, because the person we work for doesn't know what we know, and that's why they hired us.
I'm sure some people have found good employers, like you explain. And that's nice to hear. But I am actively looking at going to college for cyber security and other I.T. related fields, but am reticent because I absolutely abhor the idiocy of humanity; and it comes in spades with bad/dumb bosses.
Which is why I cited the quote that I did. It's rather resoundingly true still to this day... a good 17 years later after I left that high school where I learned the quote.
An interesting look "behind the curtain" of a high volume social networking platform. Regardless who's in charge, Twitter is an impressive service, thanks in no small part to the brilliant engineers behind the scenes.
Wonderful look at one small corner of an enterprise like Twitter. I don't think anyone believes that EVERYONE at Twitter or the other companies spend their days in Zen rooms and sipping herb tea, or whatever. But it does seem that Twitter demonstrates that there was a LOT of fluff throughout the company that many PEOPLE LIKE YOU made it possible for others to work 10 hours out of a 40-hour week. Wild guess on my part that maybe 15% of the people make it possible for 85% to live those fluff jobs. Thanks again for the insight.
Great piece! Pretty fascinating, even to a layperson. Other than personal computing for 32yrs (post DOS) I know nothing, really, BUT I know what a cache is, what one does, what servers & racks do, and I was astonished I was able to actually understand a lot of this. Even with automation services it’s not a job for today’s average gamer, obviously. I guess it’ll take some time to see if lack of staffing & tech oversight at Twitter (we assume) results in service interruptions or chaos other than in content moderation.
Glad to hear that you were part of the team that really made you work like this, essentially, encouraging you to make an effort and put it to work. In some places (read: companies), the SRE teams are treated just like an unwanted child!
I think once the new group of replacements start making changes rapidly the site will start to see instability. The biggest source of problems ime is people making changes. I think there are a lot of lessons that will need to be relearned.
Hah yeah that's a possibility for sure. I'm looking forward to trying out new things and products though. A lot of companies doing cool things out there.
Really interesting inside-out pov. Hard to see how it can continue with engineering hollowed out. Small problems slowly accumulating will be the problem. Engineering is like gardening: you need to do a little often to keep the garden looking good.
The fact that Qatar world cup started while all these changes were pushed and the web holds it talks a lot about your work. Hope you find a better job in the near future, some place where all this would be appreciated.
yeah, probably for some time it will run fine, but then software might change, so it's not running on new OS or something, changes need to be made etc. System contain some bugs. These errors will get recognized only after a certain amount of time.
I'm system administrator and i would say if i leave it will also run for a good amount of time (maybe some months) but then the problems will start to appear...
^^And when they do the people who know what they did 2 years before (or can guess quite fast since they know the system because they built it) won't be there anymore. So fixing bugs will take more time. And also fixing them in the wrong way might very well lead to other bugs. It's a bit early to put the popcorn away I'd say...
To stay online, one could just keep the current software versions. At the risk of unpatched vulnerabilities, but I think things could stay functional on "autopilot" for a year or two. The inability to support changed requirements might become a problem earlier, especially considering the current upheaval at Twitter.
Hardware side, there appear to be redundancies and automatic failover implemented, but at some point the number of defective servers will exceed the size of that reserve. At this point, the problem will become visible for Twitter users, and a decimated maintenance team might be unable to fix the problems faster than new ones crop up.
My guess is that Twitter will either suffer a technical breakdown at some point, or Elon will have to spend more money to fix things than he is saving at the moment.
I suspect Elon will let this train run, meanwhile developing "Twitter 2.0" with his own engineers, developers etc. All in a different data centre (probably on a space station, knowing Elon!)
When everything works fine, they wonder why they hired you. When everything stops working, they wonder why they hired you.
I.T. in a nutshell.
I always tell our guys that I don't care what they do, where they do it or how busy they are - I only care about the result being there and any information I might need to be given before it is too late. So I'm actually happy when people aren't overworked and double happy because I get volontary engagement if something needs to be done quickly. Live is too short to waste it by always only thinking about money.
Well, that's nice to hear. The quote I stated though is precisely why I didn't fully get into I.T. Even the I.T. teacher I was doing voluntary work for in my high school when I was younger, took the 'dumb slave driver' approach, instead of being reasonable and logical.
I would suggest that you and a few others are not the norm, but the exception to the rule.
That needs to change.
In my potentially less than humble opinion, I.T. needs to get a certain level of authority over those who we work for, because the person we work for doesn't know what we know, and that's why they hired us.
I'm sure some people have found good employers, like you explain. And that's nice to hear. But I am actively looking at going to college for cyber security and other I.T. related fields, but am reticent because I absolutely abhor the idiocy of humanity; and it comes in spades with bad/dumb bosses.
Which is why I cited the quote that I did. It's rather resoundingly true still to this day... a good 17 years later after I left that high school where I learned the quote.
An interesting look "behind the curtain" of a high volume social networking platform. Regardless who's in charge, Twitter is an impressive service, thanks in no small part to the brilliant engineers behind the scenes.
Wonderful look at one small corner of an enterprise like Twitter. I don't think anyone believes that EVERYONE at Twitter or the other companies spend their days in Zen rooms and sipping herb tea, or whatever. But it does seem that Twitter demonstrates that there was a LOT of fluff throughout the company that many PEOPLE LIKE YOU made it possible for others to work 10 hours out of a 40-hour week. Wild guess on my part that maybe 15% of the people make it possible for 85% to live those fluff jobs. Thanks again for the insight.
Jeez, imagine you weren't there and implemented those changes. Twitter would likely be dead by now. Very interesting read, thank you!
Great piece! Pretty fascinating, even to a layperson. Other than personal computing for 32yrs (post DOS) I know nothing, really, BUT I know what a cache is, what one does, what servers & racks do, and I was astonished I was able to actually understand a lot of this. Even with automation services it’s not a job for today’s average gamer, obviously. I guess it’ll take some time to see if lack of staffing & tech oversight at Twitter (we assume) results in service interruptions or chaos other than in content moderation.
Thanks glad to hear you were able to! I tried to avoid as much tech and industry terms as possible. (I think I slipped up once or twice though)
Great explanation. Thanks.
Glad to hear that you were part of the team that really made you work like this, essentially, encouraging you to make an effort and put it to work. In some places (read: companies), the SRE teams are treated just like an unwanted child!
This is a very interesting article for a Twitter user like me!
how long can twitter keep running smoothly under the current conditions, whatvis your assessment?
I think once the new group of replacements start making changes rapidly the site will start to see instability. The biggest source of problems ime is people making changes. I think there are a lot of lessons that will need to be relearned.
Thanks.
Leon Musk would be wise to re engage your services Matthew at a higher rate of pay!
Why don't you send this article to him for a start?
Hah yeah that's a possibility for sure. I'm looking forward to trying out new things and products though. A lot of companies doing cool things out there.
Really interesting inside-out pov. Hard to see how it can continue with engineering hollowed out. Small problems slowly accumulating will be the problem. Engineering is like gardening: you need to do a little often to keep the garden looking good.
Looks like Twitter is running up against the law in other jurisdictions too: https://www.rte.ie/news/courts/2022/1125/1338351-twitter-injunction/
Absent 'at will' employment contracts, it's not possible to summarily dismiss employees without due process.
"There was a few before me..."
"Well for now at least, I’m sure there’s some bugs lurking somewhere..."
It seems distinguishing between singular and plural tense is a major bug.
Bring in the robot engineers!!
The fact that Qatar world cup started while all these changes were pushed and the web holds it talks a lot about your work. Hope you find a better job in the near future, some place where all this would be appreciated.
Honestly amazing. I somehow managed to learned more from this article then my entire textbook
All I read is 80% got fired and the service is still working fine...
Can not agree more. I didn't eat today, I feel great, so I guess I won't buy groceries anymore. ;)
Your car will also continue to work fine for quite a while even if you skip all the maintenance.
yeah, probably for some time it will run fine, but then software might change, so it's not running on new OS or something, changes need to be made etc. System contain some bugs. These errors will get recognized only after a certain amount of time.
I'm system administrator and i would say if i leave it will also run for a good amount of time (maybe some months) but then the problems will start to appear...
^^And when they do the people who know what they did 2 years before (or can guess quite fast since they know the system because they built it) won't be there anymore. So fixing bugs will take more time. And also fixing them in the wrong way might very well lead to other bugs. It's a bit early to put the popcorn away I'd say...
To stay online, one could just keep the current software versions. At the risk of unpatched vulnerabilities, but I think things could stay functional on "autopilot" for a year or two. The inability to support changed requirements might become a problem earlier, especially considering the current upheaval at Twitter.
Hardware side, there appear to be redundancies and automatic failover implemented, but at some point the number of defective servers will exceed the size of that reserve. At this point, the problem will become visible for Twitter users, and a decimated maintenance team might be unable to fix the problems faster than new ones crop up.
My guess is that Twitter will either suffer a technical breakdown at some point, or Elon will have to spend more money to fix things than he is saving at the moment.
I suspect Elon will let this train run, meanwhile developing "Twitter 2.0" with his own engineers, developers etc. All in a different data centre (probably on a space station, knowing Elon!)