Git Product home page Git Product logo

rca-stories's Introduction

This is a common space where we can share:

  • mistakes we have done (which lead to some production issues), how we dealt and how we rectified
  • If you have an interesting story please create a entry and raise a PR (template is ready)
  • If you want to further discuss and share your opinions, head over to discussions

If you want to quickly share without having to raise a PR, please go ahead and fill this out: share your own rca story (It triggers an issue creation here)

Rationale

There are numerous IT companies worldwide. There are thousands of developers working on stuff, creating bugs, fixing those bugs, learning, relearning, etc. But most of that happens intra-org. Of course the big public companies with accountability share their learning and publish their major RCA (root cause analysis) but majority of them happen within the org. Most of them can't divulge those confidential information anyway.

But keeping the confidential information aside, there are a lot of learning opportunities from other's mistakes and learning for us. What if we avoid the sensitive confidential information and share those lessons for a collective learning experience.

(I am sick of this charade where you are asked to design 'book my show' and you go about adding boxes: this is api gateway, this is a cache and this is the db. Let's do some actual groundwork based learning!)

Contributing

Add your experience inside content/posts in a markdown file. (Check any existing file for the format and tagging)

Why would you contribute?

First of all it will be anonymous. If you just fill out the google form (linked top right on about section), it will trigger an issue creation on this repo which will be reviewed and added as a post. (workflow does it). Issues will be listed under my name. (since it's my github token)

Let's give it a try, shall we?

how i ended up starving a job which dealt with transaction

rca-stories's People

Contributors

sudipidus avatar

Watchers

avatar

rca-stories's Issues

I once sent a different ticket to a different user due to race condition

{
'details': We had a consumer which would listen to transaction events and based on that would issue tickets to the corresponding user. Since that was something that could be done concurrently I designed a flow to set up a threadpool and assign the tasks. Problem was that ‘data’ was part of the object’s state and not a parameter so multiple task could see and modify the ‘data’ resulting in inconsistent behavior.,
'tldr': I sent a different ticket to a different user due to race condition,
'title': I once sent a different ticket to a different user due to race condition,
}

pregrad folks didn't submit

{
"details": "pregrad folks didn't submit",
"tldr": "pregrad folks didn't submit",
"title": "pregrad folks didn't submit"
}

sherlock homes

{
"details": "surge traffic, e-commerce , just to analysis the server's data.",
"tldr": "root cause analysis",
"title": "sherlock homes"
}

sdlkjf

{
"details": "sdf",
"tldr": "sldkf",
"title": "sdlkjf"
}

data race in java concurrency

{
"details": "We had a consumer which would listen to transaction events and based on that would issue tickets to the corresponding user. Since that was something that could be done concurrently I designed a flow to set up a threadpool and assign the tasks. Problem was that ‘data’ was part of the object’s state and not a parameter so multiple task could see and modify the ‘data’ resulting in inconsistent behavior.",
"tldr": "i sent different ticket to different user because of data race",
"title": "data race in java concurrency"
}

Boring team

{
"details": "Team was not doing work on time",
"tldr": "Too boring",
"title": "Boring team"
}

blah: undefined

| What was the production issue about, what caused it and how did you solve it? | If you had to summarize and give a tldr (too long, didn't read) version of it, what would you say? | What should we title this as?
|---|---|---|---|---|---|---|
| blah | blah | blah | undefined |

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.