An open response to recent feedback Re: CPU Bottleneck

[Jirato]

This was recently shared on an external forum, and I felt it worth reposting and sharing here:

Greetings! Jirato here. Lead (and sole) developer for CLOK.

I just wanted to share that I take the negative feedback and complaints regarding CLOKs playability very seriously. I've noticed a theme in some of the replies here and in the other CLOK thread on AudioGames, suggesting"Nothing is being done about the lag," "Jirato isn't willing to fix it," "CLOK is a lost cause," And so on. I’d like to apologize if I’ve explained this poorly over the in-game chat, but I -am- indeed actively working to resolve the problem. These types of complaints aren’t being ignored, and it’s not an issue that is impossible to fix.

It might help if I explained a little bit about what's going on. The slowness and delays you are experiencing in CLOK isn’t the traditional type of “network lag" caused by a poor internet connection with the server. Rather we're dealing with what we would call a CPU bottleneck. CLOK is a single-threaded, single-process application, and it is constantly utilizing 100% of a logical CPU core. You can think of this as the entry lanes to a theme park. Some major theme parks have multiple lanes for cars to split into, and multiple ticket booths with personnel to process incoming traffic and allow them entry into the park, while some smaller theme parks may just have a single lane of traffic and a single ticket booth. This is what CLOK is like -- a single lane of traffic -- and due to the network engine that CLOK uses, we cannot simply just add new lanes of traffic and new ticket booths. You can think of it as an issue with the zoning permit initially issued when the park was first built, prohibiting additional and further road construction. A newly built theme park could use a different zoning permit and have additional lanes, but the CLOK park is, and likely always will, be limited to one lane.

But the issue isn't hopeless. To ensure the smoothest possible operation, we can reduce the amount of traffic being sent through the ticket booth by optimizing our code to use less CPU operations. Every time I work on the code, my priority in writing new systems or modifying existing systems is to make them more efficient. CLOK has gone through a very rapid stage of development over the last several years, during a long period of “alpha testing”. What alpha testing implies is a feature push – work to get as many features in the game as quickly as possible, not worrying about how optimized they may be. There are hundreds, if not thousands, of inefficiencies in the code that are causing significantly more stress on the CPU – more traffic entering and leaving the theme park. Every operation in the code is required to go through that entry booth. So are your commands, as well as the server’s responses. Often times, there's a logjam while everything waits to go through a very small opening. Thus: bottlenecking.

A more technical example: I recently performed a major overhaul of the mining system, ripping out hundreds of lines of code which had long complex nested loops that performed thousands of operations every time someone used the DIG or MINE commands. Imagine looping through each of the MUDs 7500+ rooms, multiple times, and checking the properties on every room, simply to see if it was a mine owned by the person doing the digging. Just having one player auto-digging or auto-mining would easily create considerable usage for the CPU. I gutted this all out and re-wrote it from scratch, removing all the loops and replacing it with a system that used already in-place and referenceable data so that any informational look-ups were quick and involved less operations. The finalized code for the mining system is only about 20% of what the original code was, and utilizes significantly less CPU time.

The other solution to solving our CPU bottleneck issue is to make the workers at the theme park better at their jobs so they can let more cars in and out of the park per second. To further improve the game performance, I am also in the process of transitioning to a new dedicated server. I pay for the CLOK server out of my own pocket, $170 a month, as a hobby. The current server has been leased for 5 years and was priced as a bargain bin server with an economy processor. It's great as a simple web and database server, but not sufficient for a large-scale application like CLOK. We will soon be transitioning to a brand-new server with significantly upgraded hardware. This will be costing me an additional $29 a month (Total price for the server is $2,388 a year). But CLOK is something that I love dearly and am more than willing to financially support as long as possible.

For the curious, here are the comparisons of the old and new server:

Current CPU:
Intel Xeon x3440
2.53Ghz quad core processor
CPU Mark Score: 4558

New CPU:
Intel Xeon E3-1230 v5
3.4Ghz (up to 3.8Ghz) quad core processor
CPU Mark Score: 9670

Current Memory: 8GB DDR2
New Memory: 16GB DDR4

Current Storage: 1TB Hard Disk Drive
New Storage: 256MB Business-class Solid State Drive

Current Bandwidth: 100MB dedicated connection in enterprise datacenter with multiple fully redundant 10Gbps fiber lines to major internet backbones.
New Bandwidth: 1GB dedicated connection in enterprise datacenter with multiple fully redundant 10Gbps fiber lines to major internet backbones.

For the non-technical people, the bottom line is the new hardware is over twice as powerful as the old hardware. I can't make any guarantee that this will instantly resolve all of CLOKs problems, as we still have those hundreds of inefficiencies that I will continue to resolve every day, but I do expect to see a marked performance increase under the new hardware. We are testing this new server now, will be conducting a public stress test this weekend, and should be ready to move it into production shortly after that.

In addition to all the above, we’ve also recently doubled the size of our GM team to better provide events, merchants, plot developments, and player support. I hope this shows that I am dedicated to continuing to improve your experiences in CLOK because, I promise to you that, like a theme park, CLOK is worth the wait.

[Lun]

Thank you, as always, for your hard work and dedication. It's magic, what you're doing, and often times back-end things are just that and they don't show up right away. I think complaints are a lot easier to voice than thanks, especially when something renders someone dissatisfied.

It's a monstrous task to take something, someone else's work of love, and to slowly transform it into your own ideal. I'm a terrible example of a player that's been here through thick and through thin, especially considering my constant months long hiatii but I've got nothing but respect for everyone on the staff, past and new.

So I guess what I'm trying to say in a real roundabout way is thanks. Thanks for your dedication and work around the CLOK, even when there's grumbles and groans. And that goes to the whole staff, but this one goes mostly to you, Jirato.

[ironcross32]

That's crazy, that's more than my annual property tax. Well I have seen a thing at the top of the wiki about donations, and I'd be happy to donate, but I don't use, / don't like patreon, a paypal was mentioned, but I don't see any information related to it. I asked in game and someone gave me a link to a post here on the BBS, but it wasn't relevant. So, if I could get that paypal, we could make something work here.

[Jirato]

You may make direct, one-time donations at http://www.paypal.me/clokmud

Thank you for sharing interest in supporting CLOK. I consider the server expenses the cost of my hobby, so I'm alright with them, but anything you or anyone can do to help lessen the burden would be greatly appreciated, especially this time of year.

[ironcross32]

done

[Hakon]

Awesome specs. Thank you so much for keeping this going. It makes me glad to hear you're enjoying keeping this hobby game going, and playing this game makes me happy. Win-win situation. Always a pleasure to come hang out for a while in the CLOK world. *thumbs up*

[Jirato]

Thank you for the donation and the kind words, ironcross32. I shared your note with the GM team.

Also thanks for your kind words as well, Hakon. Always happy to hear that players enjoy our efforts.