1y ago

All of Japan's Toyota Assembly Plants Shut Down for a Day Because Their Server Ran Out of Disk Space

www.reuters.com Toyota says plant shutdown last week due to server malfunction

A malfunction that shut down all of Toyota Motor's assembly plants in Japan for about a day last week occurred because some servers used to process parts orders became unavailable after maintenance procedures, the company said.

You're viewing a single thread.

115 comments

I blame lean philosophy. Keeping spare parts and redundancy is expensive so definitely don't do it...which is just rolling the dice until it comes up snake eyes and your plant shuts down.
It's the "save 5% yearly and stop trying to avoid a daily 5% chance of disaster"
Over prepared is silly, but so is under prepared.
They were under prepared.
- Lean philosophy is supposed to account for those dice-rolling moments. It's not just "keep nothing in inventory", there is supposed to be risk assessment involved.
  The problem is that leadership doesn't interpret it that way and just sees "minimizing inventory increases profit!"
  
  The problem is that leadership doesn’t interpret it that way and just sees “minimizing inventory increases profit!”
  Yep. Managers prioritize short-term gains (often personal gains, too) over the overall health of a business.
  There's also industries where the "lean" strategy is inappropriate because the given industry is one that booms in times of crisis when logistics to get "just in time" supplies go kaput due to the same catastrophe that's causing the industry to boom. Hospitals and clinics can end up in trouble like this.
  But there's other industries too--I haven't looked for it, but I'm sure there's a plethora of analysis already on what Covid did to companies and their supply chains.
  
  Why would they prioritize long term gains? Their next review is only ever at most 6 months away and they're either low-mid level and fighting for their lives every day or they're upper mgmt and can always dump stick and YOLO out, potentially with a golden parachute.
  
  In my own impression from the side of software engineering (i.e. the whole discipline rather than just "coding") this kind of thing is pretty common:
  Start with ad-hoc software development with lots of confusion, redundancy, inneficient "we'll figure it out as when we get there" and so on.
  To improve on this somebody really thinks things through and eventually a software development process emerges, something like Agile.
  There are lots of good reasons for every part of this processes but naturally sometimes the conditions are not met and certain parts are not suitable for use: the whole process is not and can never be a one size fits all silver bullet because it's way to complex and vast a discipline for that (if it wasn't you wouldn't need a process to do it with even the minimum of efficency).
  However most people using it aren't the "grand thinkers" of software engineering - software architect level types with tons of experience and who thus have seen quite a lot and know why certain elements of a process are as they are, and hence when to use them and when not to use them - and instead they're run-of-the-mill, far more junior software designers and developers, as well as people from the management side of things trying to organise a tech-heavy process.
  So you end up with what is an excellent process when used by people who know that each part tries to achieve, what's the point of that and when is it actually applicable, being used by people who have no such experience and understanding of software development processes and just use it as one big recipe, blindly following it with no real understanding and hence often using it incorrectly.
  For example, you see tons of situations where the short development cycles of Agile (aka sprints) and use cases are used without the crucial element which is actually envolving the end-users or stakeholders in the definition of the use cases, evaluation of results and even prioritization of what to do in the next sprint, so one of the crucial objectives of use cases - the discovery of the requirement details by interactive cycles with end-users where they quickly see some results and you use their feedback to fine-tune what gets done to match what they actually need (rather than the vague very high level idea they themselves have at the start of the project) is not at all achieve and instead they're little more than small project milestones that in the old days would just be entries in Microsoft Manager or some tool like that.
  This is IMHO the "problem" with any advanced systematic process in a complex domain: it's excellent in the hands of those who have enough experience and understanding of concerns at all levels to use it but they're generally either used by people without that experience (often because managers don't even recognize the value of that experience until things unexpectedly blow up) or by actual managers whose experience might be vast but is actuallly in a parallel track that's not really about dealing with the kinds of technical concerns that the process is designed to account for.
  
  it’s excellent in the hands of those who have enough experience and understanding of concerns at all levels to use it but they’re generally either used by people without that experience
  You're just saying that skilled people can do stuff better than unskilled people. This is hardly a software engineering issue. It is common in all aspects of life
  The difference with software engineering is that the field is still relatively young enough to not have figured out a working model or sets of working models (unlike farming, for example, or finance).This field is realistically 30 years old during which it continuously evolves and redefines itself so it's still not going to produce good working models.
  Agile, since you picked on it, is very difficult to implement because it specifically relies on engineering figuring out how to work and how to deliver. It's really not a model at all. It's just a set of guidelines meant to create the environment in which the figuring out happens. It's no wonder that it only works when people have the ability to figure it out.
  
  I'm not saying whatever you're trying to put in my mouth.
  In very very VERY simple terms: A software engineer with half the experience of somebody at a technical architecture level isn't half as capable a technical architect- such a person is pretty much totally incapable in that domain.
  Experience isn't linear, it's a sequence of unlocking and filling up of experience in domains which are linked but have separate concerns, with broader and broader scopes that go way beyond the mere coding, and this non-linerarity happens because it takes a while before people merelly become aware of the implications at the level at which they work of certain things outside their scope of work.
  So if you're not at the level of even being aware of how the end users of a software being developed themselves have very vague and extremelly incomplete ideas of what they need as software to help the in their own business process, then you can't even begin to see not only what's the point of certain practices around things like use cases, but even the entire need and suitability of Agile versus other development processes in a specific project and environment, so you're not at all qualified to decide which parts of that to do and which not to do in the specific situation of your specific project, or even if Agile is the right choice.
  People who don't even know about the forms of requirements gathering in different environments can't even begin to evaluate the suitability for their environment of a Process such as Agile which was designed mostly to address the "fast changing requirements" project situations, which are the product of various weakness in requirements gathering and/or fast changing business needs, which at the development side snowball into massive problems when long-development-cycle processes such as waterfall are used (for example when supposedly "done projects" do not produced something that matches stakeholder needs, hence end up having to be "fixed" so late in the process that it massivelly disrupts the software at a design and even architectural level, introducing massive weaknesses in the code base and code spaghettization, hence bugs and maintenability nightmares).
- I work in a manufacturing company that was owned by the founder for 50 years until about 4 years ago when he retired. He disagreed with a lot of the ideas behind lean manufacturing so we had like 5 years worth of inventory sitting in our warehouse.
  When the new management came in, there was a lot of squawking about inefficiency, how wasteful it was to keep so much raw material on the shelf, and how we absolutely needed to sell it off or get rid of it.
  Then a funny little thing happened in 2020.
  Suddenly, we were the only company in our industry still churning out product. Other companies were calling us, desperate to buy our products or even just our raw material. We saw MASSIVE growth the next two years and came out of the pandemic better than ever. And it was mostly thanks to the old owners view that "Just In Time" manufacturing was BS.
  
  Cool story, but a once every 150 years pandemic is hardly a good reason to keep wasting money on storing stuff. A fire or a flood was much more likely to wipe it all out in 50 years.
  Even in your anecdote the owner never actually benefited from the extra costs.
  Depending on what you're producing costs to maintain extra inventory of raw materials can be massive and for the company the size of Toyota, multiply that by million.
  
  You're not wrong, but the real answer is somewhere in between.
  
  Even in your anecdote the owner never actually benefited from the extra costs.
  Imagine doing something or having a life/business philosophy that doesn't exist for your own soul benefit, and exists maybe for the benefit of others.
  
  Oh man. I work for an OEM and have almost the same story except the CEO didn't retire so our shelves were never bare. A large part of our sales last year was because our competition couldn't source parts and we could. Also since they have a skeleton engineering crew they couldn't figure out how to improvise (at least this is what our salespeople are claiming).
  2022 I spent 3 hours a day, every single working day, just making ECNs for parts out of stock.
  Lean JIT is such fucking crap. When a customer line goes down they are bleeding who knows how many millions a day. They want a solution right freaken now. So they call the OEM and we tell them that part that died we have right here on the shelf and we can overnight it.
  But you know how you really know that JIT Evengelical types are full of it? Talk to any of them now and they won't say something like "look, normally it works but no one could have predicted this" they will double down on it and say it works but was implement incorrectly. It isnt "true" JIT.
  Many years ago I read a sentence that I use every single week of my life "all ideology can do is reassert itself endlessly". When I hear crap like that, how X works if and only if it is true-X I know I am dealing with ideology. When I hear that X works under very specific situations and only gives specific results I know I am dealing with facts. Yes if the customer is never in a rush, there isn't a world shaking event, and you are running out of capacity for storage JIT might be an answer.

115 comments