An open response to recent feedback Re: CPU Bottleneck
Posted: Fri Oct 06, 2017 6:15 am
This was recently shared on an external forum, and I felt it worth reposting and sharing here:
Greetings! Jirato here. Lead (and sole) developer for CLOK.
I just wanted to share that I take the negative feedback and complaints regarding CLOKs playability very seriously. I've noticed a theme in some of the replies here and in the other CLOK thread on AudioGames, suggesting"Nothing is being done about the lag," "Jirato isn't willing to fix it," "CLOK is a lost cause," And so on. I’d like to apologize if I’ve explained this poorly over the in-game chat, but I -am- indeed actively working to resolve the problem. These types of complaints aren’t being ignored, and it’s not an issue that is impossible to fix.
It might help if I explained a little bit about what's going on. The slowness and delays you are experiencing in CLOK isn’t the traditional type of “network lag" caused by a poor internet connection with the server. Rather we're dealing with what we would call a CPU bottleneck. CLOK is a single-threaded, single-process application, and it is constantly utilizing 100% of a logical CPU core. You can think of this as the entry lanes to a theme park. Some major theme parks have multiple lanes for cars to split into, and multiple ticket booths with personnel to process incoming traffic and allow them entry into the park, while some smaller theme parks may just have a single lane of traffic and a single ticket booth. This is what CLOK is like -- a single lane of traffic -- and due to the network engine that CLOK uses, we cannot simply just add new lanes of traffic and new ticket booths. You can think of it as an issue with the zoning permit initially issued when the park was first built, prohibiting additional and further road construction. A newly built theme park could use a different zoning permit and have additional lanes, but the CLOK park is, and likely always will, be limited to one lane.
But the issue isn't hopeless. To ensure the smoothest possible operation, we can reduce the amount of traffic being sent through the ticket booth by optimizing our code to use less CPU operations. Every time I work on the code, my priority in writing new systems or modifying existing systems is to make them more efficient. CLOK has gone through a very rapid stage of development over the last several years, during a long period of “alpha testing”. What alpha testing implies is a feature push – work to get as many features in the game as quickly as possible, not worrying about how optimized they may be. There are hundreds, if not thousands, of inefficiencies in the code that are causing significantly more stress on the CPU – more traffic entering and leaving the theme park. Every operation in the code is required to go through that entry booth. So are your commands, as well as the server’s responses. Often times, there's a logjam while everything waits to go through a very small opening. Thus: bottlenecking.
A more technical example: I recently performed a major overhaul of the mining system, ripping out hundreds of lines of code which had long complex nested loops that performed thousands of operations every time someone used the DIG or MINE commands. Imagine looping through each of the MUDs 7500+ rooms, multiple times, and checking the properties on every room, simply to see if it was a mine owned by the person doing the digging. Just having one player auto-digging or auto-mining would easily create considerable usage for the CPU. I gutted this all out and re-wrote it from scratch, removing all the loops and replacing it with a system that used already in-place and referenceable data so that any informational look-ups were quick and involved less operations. The finalized code for the mining system is only about 20% of what the original code was, and utilizes significantly less CPU time.
The other solution to solving our CPU bottleneck issue is to make the workers at the theme park better at their jobs so they can let more cars in and out of the park per second. To further improve the game performance, I am also in the process of transitioning to a new dedicated server. I pay for the CLOK server out of my own pocket, $170 a month, as a hobby. The current server has been leased for 5 years and was priced as a bargain bin server with an economy processor. It's great as a simple web and database server, but not sufficient for a large-scale application like CLOK. We will soon be transitioning to a brand-new server with significantly upgraded hardware. This will be costing me an additional $29 a month (Total price for the server is $2,388 a year). But CLOK is something that I love dearly and am more than willing to financially support as long as possible.
For the curious, here are the comparisons of the old and new server:
Current CPU:
Intel Xeon x3440
2.53Ghz quad core processor
CPU Mark Score: 4558
New CPU:
Intel Xeon E3-1230 v5
3.4Ghz (up to 3.8Ghz) quad core processor
CPU Mark Score: 9670
Current Memory: 8GB DDR2
New Memory: 16GB DDR4
Current Storage: 1TB Hard Disk Drive
New Storage: 256MB Business-class Solid State Drive
Current Bandwidth: 100MB dedicated connection in enterprise datacenter with multiple fully redundant 10Gbps fiber lines to major internet backbones.
New Bandwidth: 1GB dedicated connection in enterprise datacenter with multiple fully redundant 10Gbps fiber lines to major internet backbones.
For the non-technical people, the bottom line is the new hardware is over twice as powerful as the old hardware. I can't make any guarantee that this will instantly resolve all of CLOKs problems, as we still have those hundreds of inefficiencies that I will continue to resolve every day, but I do expect to see a marked performance increase under the new hardware. We are testing this new server now, will be conducting a public stress test this weekend, and should be ready to move it into production shortly after that.
In addition to all the above, we’ve also recently doubled the size of our GM team to better provide events, merchants, plot developments, and player support. I hope this shows that I am dedicated to continuing to improve your experiences in CLOK because, I promise to you that, like a theme park, CLOK is worth the wait.