Historical note: This article documents a performance issue encountered in a .NET Framework/WCF application around 2009. The connection-management setting described below applies to particular HTTP-based configurations and should not be assumed to affect every WCF binding or modern .NET application.
Chasing the Wind
Previously, I was working on a C# application to performance test a new mainframe replacement system.
The application dispatches units of work through an Oracle Service Bus (OSB) to C++ services on an AIX box. It uses the Supervisor Pattern where a single dispatcher, using XMPP, sends work to multiple workers on different PCs. The multi-threaded workers connect through WCF to the OSB.
The use of XMPP was to get around perceived bottlenecks with WCF. It was blazingly fast.
Although the XMPP dispatcher could distribute work well beyond the required throughput, the workers were managing only a few transactions per second through WCF to the OSB.
Start with the Simplest Solution
After eliminating everything except the simplest thing, I discovered that there is a per-AppDomain outbound connection limitation default of two for each remote server. It can be adjusted via the connectionManagement configuration section. This simple change was all that was required:
<configuration>
<system.net>
<connectionManagement>
<add address="*" maxconnection="30" />
</connectionManagement>
</system.net>
</configuration>Solving the Impossible
The performance target was 3,000 transactions per second, but throughput was only part of the challenge. I also needed to replay historical betting transactions in their original sequence.
I experimented with slight delays, batched feeds, and scheduling transactions using the time deltas between historical events. Each approach presented a different trade-off between throughput and timing fidelity.
The fundamental problem was that I controlled the producer, not the consumers. Once transactions passed through the Oracle Service Bus, I had no control over when the downstream C++ services picked them up, processed them, or committed their results. Nor could I control the effects of network latency, timeouts, resource contention, or other unpredictable delays.
That was perhaps the most important lesson of the project: sometimes the problem isn't the algorithm or the implementation, but the boundaries within which you're being asked to solve it.
And sometimes good enough is enough.