Notes, Quotes, and Thoughts from Release It
./john

Beware this necessary evil. Every integration point will eventually fail in some way, and you need to be prepared for that failure.

Prepare for the many forms of failure. Integration point failures take several forms, ranging from various network errors to semantic errors. You will not get nice error responses delivered through the defined protocol; instead, you'll see some kind of protocol violation, show response, or outright hang.

Know when to open up abstractions. Debugging integration point failures usually requires peeling back a layer of abstraction. Failures are often difficult to debug at the application layer because most of them violate the high-level protocols. Packet sniffers and other network diagnostics can help.

Failures propagate quickly. Failure in a remote system quickly because your problem, usually as a cascading failure when your code isn't defensive enough.

Apply patterns to avert integration point problems. Defensive programming via Circuit Breaker, Timeouts, Decoupling Middleware, and Handshaking will all help you avoid the dangers of integration points.

Recognize that one server down jeopardizes the rest. A chain reaction happens because the death of one server makes the others pick up the slack. The increased load makes them more likely to fail. A chain reaction will quickly bring an entire layer down. Other layers that depend on it must protect themselves, or they will go down in a cascading failure.

Hunt for resource leaks. Most of the time, a chain reaction happens when your application has a memory leak. As one server runs out of memory and goes down, the other servers pick up the dead one's burden. the increased traffic means they leak memory faster.

Hunt for obscure timing bugs. Obscure race conditions can also be triggered by traffic. Again, if one server goes down to a deadlock, the increased load on the others makes them more likely to hit the deadlock too.

Use Autoscaling. In the cloud, you should create health checks for every autoscaling group. The scaler will shut down instances that fail their health checks and start new ones. As long as the scaler can react faster than the chain reaction propagates, your service will be available.

Defend with Bulkheads. Partitioning servers with Bulkheads can prevent chain reactions from taking out the entire service - though they won't help the callers of whichever partition does go down. Use Circuit Breaker on the calling side for that.

Stop cracks from jumping the gap. A cascading failure occurs when cracks jump from one system or layer to another, usually because of insufficiently paranoid integration points. A cascading failure can also happen after a chain reaction in a lower layer. Your system surely calls out to other enterprise systems; make sure you can stay up when they go down.

Scrutinize resource pools. A cascading failure often results from a resource pool, such as a connection pool, that gets exhausted when none of its calls return. The threads that get the connections block forever; all other threads get blocked waiting for connections. Safe resource pools always limit the time a thread can wait to check out a resource.

Defend with Timeouts and Circuit Breaker. A cascading failure happens after something else has already gone wrong. Circuit Breaker protects your system by avoiding calls out to the troubled integration point. Using Timeouts ensures that you can come back from a call out to the troubled point.

Users consume memory. Each user's session requires some memory. Minimize that memory to improve your capacity. Use a session only for caching so you can purge the session contents if memory gets tight.

Users do weird, random things. Users in the real world do things that you won't predict (or sometimes understand). If there's a weak spot in your application, they'll find it through sheer numbers. Test scripts are useful for functional testing but too predictable for stability testing. Look into fuzzing toolkits, property-based testing, or simulation testing.

Malicious users are out there. Become intimate with your network design; it should help avert attacks. Make sure your systems are easy to patch - you'll be doing a lot of it. Keep your frameworks up-to-date, and keep yourself educated.

Users will gang up on you. Sometimes they come in really, really big mobs. When Taylor Switch tweets about your site, she's basically pointing a sword at your servers and crying, "Release the legions!" Large mobs can trigger hangs, deadlocks, and obscure race conditions. Run special stress tests to hammer deep links or hot URLs.

Recall that the Blocked Threads antipattern is the proximate cause of most failures. Application failures nearly always relate to the Blocked Threads in one way or another, including the ever-popular "gradual slowdown" and "hung server." The Blocked Threads antipattern leads to Chain Reactions and Cascading Failures antipattern.

Scrutinize resource pool. Like Cascading Failures, the Blocked Threads antipattern usually happens around resource pools, particularly database connection pools. A deadlock in the database can cause connections to be lost forever, and so can incorrect exception handling.

Use proven primitives. Learn and apply safe primitives. It might seem easy to roll your own producer/consumer queue: it isn't. any library of concurrency utilities has more testing than your newborn queue.

Defend with Timeouts. You cannot prove that your code has no deadlocks in it, but you can make sure that no deadlock lasts forever. Avoid infinite waits in function calls; use a version that takes a timeout parameter. Always use timeouts, even though it means you need more error-handling code.

Beware the code you cannot see. All manner of problems can lurk in the shadows of their-party code. Be very wary. Test it yourself. Whenever possible, acquire and investigate the code for surprises and failure modes. You might also prefer open source libraries to closed source for this very reason.

Keep the lines of communication open. Self-denial attacks originate inside your own organization, when people cause self-inflected wounds by creating their own flash mobs and traffic spikes. You can aid and abet these marketing efforts and protect your system at the same time, but only if you know what's coming. Make sure nobody sends mass emails with deep links. Send mass emails in waves to spread out the peak load. Create static "landing zone" page for the first click from these offers. Watch out for embedded session IDs in URLs.

Protect shared resources. Programming errors, unexpected scaling effects, and shared resources all create risks when traffic surges. Watch out for Fight Club bugs, where increased front-end load causes exponentially increasing back-end processing.

Expect rapid redistribution of any cool or valuable offer. Anybody who thinks they'll release a special deal for limited distribution is asking for trouble. There's no such thing as limited distribution. Even if you limit the number of times a fantastic deal can be redeemed, you'll still get crushed with people hoping beyond hope that they, too, can get a PlayStation Twelve for $99.

Examine production versus QA environments to spot Scaling Effects. You get bitten by Scaling Effects when you move from small one-to-one development and test environments to full-sized production environments. Patterns that work fine in small environments or one-to-one environments might slow down or fail completely when you move to productions sizes.

Watch out for point-to-point communication. Point-to-point communication scales badly, since the number of connections increases as the square of the number of participants. Consider how large your system can grow while still using point-to-point connections - it might be sufficient. Once you're dealing with tens of servers, you will probably need to replace it with some kind of one-to-many communication.

Examine server and thread counts. In development and QA, your system probably looks like one or two servers, and so do all the QA versions of other systems you call. In production, the ratio might be more like ten to one instead of one to one. Check the ratio of front-end to backend servers, along with the numbers of threads each side can handle in production, compared to QA.

Observe near Scaling Effects and users. Unbalanced Capacities is a special case of Scaling Effects: one side of a relationship scales up much more than the other side. A change in traffic patterns - season, market-drive, or publicity-driven - can cause a usually benign front-end system to suddenly flood a back-end system, in much the same way as a hot Reddit post or celebrity tweet causes traffic to suddenly flood websites.

Virtualize QA and scale it up. Even if your production environment is a fixed size, don't let your QA languish at a measly pair of servers. Scale it up. Try test cases where you scale the caller and provider to different ratios. You should be able to automate this through your data center automation tools.

Stress both sides of the interface. If you provide the back-end system, see what happens if it suddenly gets ten times the highest-ever demand, hitting the most expensive transaction. Does it fail completely? Does it slow down and recover? If you provide the front-end system, see what happens if calls to the bend end stop responding or get very slow.

Dogpiles force you to spend too much to handle peak demand. A dogpile concentrates demands. It requires a higher peak capacity than you'd need if you spread the surge out.

Use random clock slew to diffuse the demand. Don't set all your cron jobs for midnight or any other on-the-hour time. Mix them up to spread the load out.

Use increasing backoff times to avoid pulsing. A fixed retry interval will concentrate demand from callers on that period. Instead, use a backoff algorithm so different callers will be at different points in their backoff periods/

Ask for help before causing havoc. Infrastructure management tools can make very large impacts very quickly. Build limiters and safeguards into them so they won't destroy your whole system at once.

Beware of lag time and momentum. Actions initiated by automation take time. That time is usually longer than a monitoring interval, so make sure to account for some delay in the system's response to action.

Beware of illusions and superstitions. Control systems sense the environment, but they can be fooled. They compute an expected state and a "belief" about the current state. Either can be mistaken.

Slow Responses trigger Cascading Failures. Upstream systems experiencing Slow Responses will themselves slow down and might be vulnerable to stability problems when the response times exceed their own timeouts.

For websites, Slow Responses cause more traffic. Users waiting for pages frequently hit the Reload button, generating even more traffic to your already overloaded system.

Consider Fail Fast. If your system tracks its own responsiveness, then it can tell when it's getting slow. consider sending an immediate error response when the average response time exceeds the system's allowed time (or at the very least, when the average response time exceeds the caller's timeout!).

Hunt for memory leaks or resource contention. Contention for an inadequate supply of database connections produces Slow Responses. Slow Responses also aggravate that contention, leading to a self-reinforcing cycle. Memory leaks cause excessive effort in the garbage collector, resulting in Slow Responses. Inefficient low-level protocols can cause network stalls, also resulting in Slow Responses.

Use realistic data volumes. Typical development and test data sets are too small to exhibit this problem. You need production-sized data sets to see what happens when your query returns a million rows that turn into objects. As a side benefit, you'll also get better information from your performance testing when you use production sized test data.

Paginate at the front end. Build pagination details into your service call. The request should include a parameter for the first item and the count. The reply should indicate (roughly) how many results there are.

Don't rely on the data producers. Even if you think a query will never have more than an handful of results, beware: it could change without warning because of some other part of the system. The only sensible numbers are "zero," "one," and lots," so unless you query selects exactly one row, it has potential to return too many. Don't rely on the data producers to create a limited amount of data. Sooner or later, they'll go berserk and fill up a table for no reason, and then where will you be?

Put limits into other application-level protocols. Service calls, RMI, DCOM, XML-RPC, and any other kind of request/reply call are vulnerable to returning huge collections of objects, thereby consuming too much memory.

Apply Timeouts to Integration Points, Blocked Threads, and Slow Responses. The Timeouts pattern prevents calls to Integration Points from becoming Blocked threads. thus, timeouts avert Cascading Failures.

Apply Timeouts to recover from unexpected failures. When an operation is taking too long, sometimes we don't care why… we just need to give up and keep moving. The Timeouts pattern lets us do that.

Consider delayed retires. Most of the explanations for timeout involve problems in the network or the remote that won't be resolved right away. Immediate retries are liable to hit the same problem and result in another timeout. That just makes the user wait even longer for her error message. Most of the time, you should just queue the operation and retry it later.

Don't do it if it hurts. Circuit Breaker is the fundamental pattern for protecting your system from all manner of Integration Points problems. When there's a difficulty with Integration points, stop calling it!

Use together with Timeouts. Circuit Breaker is good at avoiding calls when Integration Points has a problem. The Timeouts pattern indicates that there's a problem in Integration Points.

Expose, track, and report state changes. Popping a Circuit Breaker always indicates something abnormal. It should be visible to Operations. It should be reported, recorded, trended, and correlated.

Save part of the ship. The Bulkheads pattern partitions capacity to preserve partial functionality when bad things happen.

Pick a useful granularity. You can partition thread pools inside an application, CPUs in a server, or servers in a cluster.

Consider Bulkheads particular with shared services models. Failures in service-oriented or microservice architectures can propagate very quickly. If your service goes down because of a Chain Reaction, does the entire company come to a halt? Then you'd better put in some Bulkheads.

Avoid fiddling. Human intervention leads to problems. Eliminate the need for recurring human intervention. Your system should run for at least a typical deployment cycle without manual disk cleanups or nightly restarts.

Purge data with application logic. DBAs can create scripts to purge data, but they don't always know how the application behaves when data is removed. Maintaining logical integrity, especially if you use an ORM tool, requires the application to purge its own data.

Limit caching. In-memory caching speeds up applications, until it slows them down. Limit the amount of memory a cache can consume.

Roll the logs. Don't keep an unlimited amount of log files. Configure log file rotation based on size. If you need to retain them for compliance, do it on a non-production server.

Avoid Slow Responses and Fail Fast. If your system cannot meet its SLA, inform callers quickly. Don't make them wait for an error message, and don't make them wait until they time out. That just makes your problem into their problem.

Reserve resources, verify Integration Points early. In the them of "don't do useless work," make sure you'll be able to complete the transaction before you start. If critical resources aren't avaiable - for example, a popped Circuit Breaker on a required callout - then don't waste work by getting to that point. The odds of it changing between the beginning and the middle of the transaction are slim.

Use for input validation. Do basic user input validatoin even before your reserve resources. Don't bother checking out a databse connection, fetching domain objects, populating them, and calling validate() just to find out that a required parameter wasn't entered.

Crash components to save systems. It may seem counterintuitive to create system-level stability through component-level instability. Even so, it may be the best way to get back to a known good state.

Restart fast and reintegrate. The key to crashing well is to get back up quickly. Otherwise you risk loss of service when too many components are bouncing. Once a component is back up, it should be reintegrated automatically.

Isolate components to crash independently. Use Circuit Breakers to isolate callers from components that crash. use supervisors to determine what the span of restarts should be. Design your supervision tree so that crashes are isolated and don't affect unrelated functionality.

Don't crash monoliths. Large processes with heavy runtimes or long startups are not the right place to apply this pattern. applications that couple many features into a single process are also a poor choice.

Create cooperative demand control. Handshaking between a client and a server permits demand throttling to serviceable levels. Both the client and the server must be built to perform handshaking. Most common application-level protocols do not perform handshaking.

Consider health checks. Use health checks in clustered or load-balanced services as a way for instances to handshake with the load balancer.

Build handshaking into your own low-level protocols. If you create your own socket-based protocol, build handshaking into it so that the endpoints can each inform the other when they are not ready to accept work.

Emulate out-of-spec failures. Calling real aplications lets you test only those errors that the real application can deliberately produce. A good test harness lets you simulate all sorts of messy, real-world failures.

Stress the caller. The test harness can produce slow responses, no responses, or garbage responses. then you can see how your application reacts.

Leverage shared harnesses for common failures. You don't necessarily need a separate test harness for each integration point. A "killer" server can listen to several ports, creating different failure modes depending on which port you connect to.

Supplement, don't replace, other testing methods. The Test Harness pattern augments other testing methods. It does not replace unit tests, acceptance tests, penetration tests, and so on. Each of those techniques help verify functional behavior. A test harness helps verify "nonfunctional" behavior while maintaining isolation from the remote systems.

Decide at the last responsible moment. Other stability patterns can be implemented without large-scale changes to the design or architecture. Decoupling middleware is an architecture decision. It ripples into every part of the system. This is one of the nearly irreversible decisions that should be made early rather than late.

Avoid many failure modes through total decoupling. The more fully you decouple individual servers, layers, and applications, the fewer problems you will observe with Integration Points, Cascading Failures, Slow Responses, and Blocked Threads. You'll find that decoupled applications are also more adaptable, since you can change any of the participants independently of others.

Learn many architectures, and choose among them. Not every system needs to look like a three-tier application with a relational database. Learn many architectural styles, and select the best architecture for the problem at hand.

You can't out-scale the world. No matter how large your infrastructure or how fast you can scale it, the world has more people and devices than you can support. If your service is exposed to uncontrolled demand, then you need to be able to shed load when the world goes crazy on you.

Avoid slow responses using Shed Load. Creating slow responses is being a bad citizen. Keep your response times under control rather than getting so slow that callers time out.

Use load balancers as shock absorbers. Individual instances can report HTTP 503 to get some breathing room. Load balancers are good at recycling connections very quickly.

Back Pressure creates safety by slowing down consumers. Consumers will experience slowdowns. The only alternative is to let them crash the provider.

Apply Back Pressure with a system boundary. Across boundaries, look at load shedding instead. This is especially true when the Internet at large is your user base.

Queues must be finite for response times to be finite. You only have a few options when a queue is full. All of them are unpleasant: drop data, refuse work, or block. Consumers must be careful not to block forever.

Slow things down to allow intervention. When things are about to go off the rails, we often find automation tools pushing the throttle to its limit. Humans are better at situational thinking, so we need to create opportunities for us to intervene.

Apply resistance in the unsafe direction. Some actions are inherently unsafe. Shutting down, deleting, blocking things…these are all likely to interrupt service. Automation will make them go fast, so you should apply Governor to provide humans with time to intervene.

Consider a response curve. Actions may be safe within a defined range. Outside that range they should encounter increasing resistance by slowing down the rate by which they can occur.