« Non-Physical Land Racer on Phlox
Sep
14
Internet scale architecture and InWorldz
- Cari.Net,
- infrastructure
by admin
Whether it is dedicated servers, virtual servers/Virtual Machines, or a cloud computing host, there is no escaping that as a business grows on the internet they will need to utilize more and more resources toward supporting the core of their operations. These core services need to stay up and running at all costs, and need to be able to scale out depending on the demand. Any failures in these core services will cause total downtime.
Without proper support of core business processes a business on the internet is doomed to fail under high growth, or temporary high load scenarios. To application engineers this doesn’t become apparent until they scale out past the single server level, or have a write scaling issue that a traditional database cluster can’t help with.
With that said, I present a rough diagram of the InWorldz core architecture for Q4 2011 and into the future.
Important services to our customers include:
- Asset Cluster: (4 servers) Access to assets which represent all the textures, sounds, scripts and other low level data in world.
- Inventory Cluster: (4 servers) Access to the structure of your inventory, the way your folders and items are laid out. The name of, and permissions on your items in inventory.
- Region Data Cluster: (3 servers) When your region restarts, where all the “stuff” comes from to bring it back and where all the changes are stored.
- Simulation: (lots of servers with Virtual Machine partitions) The pool of servers that powers the actual simulation on your region. These are the machines you actually connect with when you log in.
You’ll note there are 13 servers that power just the InWorldz, LLC core. The first cluster in the diagram holds in world assets. Our asset systems run custom written software that help to load balance and make sure assets are available at high speeds to our customers. These assets are stored in a format that is fast to write and easy to back up.
The inventory cluster will be running a NoSQL solution to support heavy reads and writes as well as a three server replication level. This means we can lose up to two servers out of the cluster simultaneously and not have data loss that would result in having to restore a backup. As inventory requirements grow, so does this cluster simply by adding servers and maintaining a data balance. This is the same type of technology used by larger internet sites such as FaceBook and Twitter and it will be employed at InWorldz to provide you with fast and redundant inventory access.
The region data cluster will utilize a high performance SAN and a sharded MySQL installation on a high availability virtual machine software platform.
All this is meaningless without good hardware and a good host.
All of these servers run on our own gigabit network with the core of the datacenter running at 10 gb/s. All of our servers have dual power supplies connected to fully redundant batteries and backup generators and we have a scalable connection to the internet.
Recently San Diego had a major power outage that affected 1.4 million of SDG&E’s customers. Our host Cari.Net is one of those customers. But we didn’t notice at all, and didnt have any hiccups in service. This is because the power infrastructure at Cari.Net is second to none with redundancy all the way to the supply, two huge generators that keep everything running, and a fuel delivery service to keep those generators pumping until the power comes back on. (for more details click here)
We are proud to have a partner working with us that feels the same way we do about quality.
Why does InWorldz need all this “stuff”? Because we’re thinking big, not small. We want to power your dreams, not your catnaps. We’re looking into the future where we see a need for supporting vast concurrency on an interconnected grid with thousands of concurrent visitors. We’re looking into a future where virtual world technology is so important to some that it can not bear frequent or long lived downtime and outages.
Could we run the grid much cheaper? Yes we could. Would it sacrifice quality? Absolutely, and unacceptably so. As a company that is in this for the long haul we have no intention of trying to be the cheapest flavor of the month. We’re putting serious thought and investment into our network, our partners, and our people to ensure we can grow. We believe we run at a price point that is sustainable for growth and allows us to heavily invest in R&D and quality of service for the future.
There will be bumps in the road while we work towards this ideal, and it is no doubt a long road. There will be continuous learning and changing but in the end a persistent vision of a reliable, dependable world will be realized. At InWorldz, we don’t treat virtual worlds as games. We know they represent far more than that to the people that use them.
This post has no tag
Leave a Reply Cancel reply
Your email address will not b
11 comments
5 pings
Wayfinder says:
September 14, 2011 at 10:12 pm (UTC 0)
Reply
Since before Elf Clan joined Inworldz, I’ve had the highest respect for the opinions of the Founders, namelly: customer and foundation comes first. That was one of my peeves about (some other companies)– they were playing with toys when the ship’s engine was faltering and icebergs tearing holes in the hulls.
The path to a secure future is stability. We all love “features” and some of us love “toys”… but those features and toys do no good if the floor is crumbling out from under us. It does no good to have paintings hanging on the walls when the walls themselves keep falling down.
When I decided to invest in Inworldz and began the process of bringing a large group over to this grid, there were half a dozen things that prompted that decision. Top of the list was “We’re doing our own dev work and re-writing unstable code”. We appreciated (and had to have) the ability to handle group, the feature of complex inventory, estate tools. But what cinched the deal, the thing that made us decide on Inworldz, was your recognition that stable foundation was the key to future growth… and your insistence on working on that foundation even when some folks clamored for this or that feature.
That dedication to structure, to stability, to making sure things work from the core level… is what told me that Inworldz was not only in for the long haul, but that the product would actually arrive at the intended destination. Region stability, asset server stability, the ability to handle concurrency… those are the things at which all other companies had failed. Once those things are down pat and regular maintenance / growth planned, once it is realized the foundation system itself is ready for prime time… that is when toys and play pretties can be added.
I have nothing but applause for the way that Inworldz has recognized how extremely important the “back end” is to a grid… and the attention your company has paid t that back end, even sometimes at the sacrifice of attracting new users who want automobiles and planes before the roadways and runways are in place. The result: a system in which major, years-long, inherited bugs are getting fixed at a surprisingly rapid pace.
The toys will come eventually. For right now, it’s really nice to find out the toilet works, the roof isn’t leaking, and the lights come on when we flip a switch. More importantly… it’s nice to know they will still do so when 30,000 people try to do so at the same time. : )
Anyraya Braveheart says:
September 15, 2011 at 10:44 pm (UTC 0)
Reply
WOW!!!
Apart from the fact that most of what is said above is chinese to me, I can only applaud and keep faith in what is being achieved here.
Me too, I found out from the first day I joined a year and a half ago that this was THE GRID to be and decided to invest as well.
I am proud of you guys, very proud to be part of this word..
Keep up this wonderful work…
Lady Sakai says:
September 16, 2011 at 11:49 am (UTC 0)
Reply
As a long time Region / Estate owner I know first hand how important this is… the regular resident do not. They just see the result of when its NOT… re SL missing inventories and missing rezzed stuff on restarts. Can count the times that has happends in SL to me and others (no Im not putting down SL but that is the first VW of its kind so Im comparing)
I know merchants both big and small that is sometimes flodded with messages of missing inventory or thngs that dont get delivered.
At the core of these issues are the core of the grid.
Its imperativ that that works and works perfectly (well as perfectly as tech cn these days)
Its also imperativ that “owners” of different grids ae aware of this and work tireless to forsee and prevent, best to their abilities, these issues.
Raegan Viper says:
September 19, 2011 at 5:18 pm (UTC 0)
Reply
I also applaud the hard work of the founders. I can’t say anything different than those above me have already said. So I’ll simply say ditto and a huge thank you for your desire to make this a place of enjoyment and working toward putting a solid ground under our feet.
soror Nishi says:
September 20, 2011 at 1:55 pm (UTC 0)
Reply
I am convinced that SL’s main problem is that they cobbled together some cheap stuff at the beginning and have been using ‘work-arounds’ ever since.
As with all architecture, in any real setting, the provision of substantial foundations and services is what makes the difference between a development and a slum.
I am very happy to make my small monthly financial contribution to this scheme as I know it is money well spent.
Keep up the “let’s do it properly” attitude, it will save you sleepless nights in the future.
Caralitissa says:
September 29, 2011 at 2:56 pm (UTC 0)
Reply
To defend SL a bit: I think they could not imagine how fast they would have to grow – there was an incredible run on them, I can remember that thousands of new members registered daily.
That doesn’t apologise the mistakes they make since then …
All I can say is: I am happy that I found InWorlz – to me it’s the best of all grids (and I know some) and this solid ground under our feet makes it very special, also the policy of the founders to increase stability before increasing features – and to keep it affordable!
Applause and many thanks to the founders!
admin says:
October 16, 2011 at 4:41 am (UTC 0)
Reply
Definitely agreed Caralitissa. Growing faster than you’re prepared for can definitely force a business to cut corners. That’s why we think it’s very important to not do any self promotion until we’re absolutely ready.
Thank you for the feedback!
Seren Seraph says:
October 29, 2011 at 10:01 pm (UTC 0)
Reply
One think I am curious about is why there was a need to build all of this yourself versus using some of the cloud services out there. I have read that Amazon EC2 and associated products include much of the capability you mention including a reduntantly backed up distributed, scalable MySQL solution. I am curious because I am making buy/build decisions in a quite different product space that has some similar needs in scalability and dependability. It would be great to get the opinion/wisdom of those that have examined this deeply in practice.
admin says:
October 30, 2011 at 3:26 am (UTC 0)
Reply
It comes down to cost, reliability, and control.
Costs: We dont have to pay for every transfer, every byte in and out of the network, or every transaction to disks and clusters. We have unlimited bandwidth, with no surprise charges. We can have persistent, always on oceans, mainland, straits, and welcome areas without incurring additional costs. As we grow larger the billions of requests made on core infrastructure will not start to nickel and dime the core business.
Reliability: Not talking about downtime necessarily, but more the ability to control and maintain latency, internal available bandwidth, and server responsiveness. If we start to see local switching fabric get saturated we can subnet, add additional switching/routing etc.. If any part of the network misbehaves we dont have to worry about being degraded but still “within tolerances”. We can act quickly to being things back to acceptable levels.
Control: A group of dedicated engineers that maintain the hardware, and because of our joint venture, actually care about our business personally. Our success is their success, so there is a great motivation to make sure we’re up, running, and operating at the best level we can. This is an environment that doesn’t tolerate large variances in any systems without customers noticing.
Fleure Homewood says:
November 4, 2011 at 3:48 pm (UTC 0)
Reply
i LOVE you guys! Thanks for all your hard work, and oh, yes, your genius too.
kathi Blindside says:
November 10, 2011 at 6:27 pm (UTC 0)
Reply
i am not a computer geek….but i am learning more here in inworldz than i did in sl..i like to use my mind to create things with my writing and with building…in inworldz there are many wonderful places i can feel the stories and songs by their beauty of the places i have visited since coming to inworldz….in sl i could not build my own creations because the lack of prims …in inworldz i have more than enough prims to work with and i am building my ice mountain ..which starts at 1800 m and so far , to about 2000m give or take a few m’s.
i want to thank you for giving me the opportunity to enhance my mind and ability create my own dream world…i am 56 yrs old and have been on line for only 1 yr. &3.5 months..i am not computer wise but i love being in inworldz and sl ..keep up with the good work …we disabled ppl need a place to inter act w/ppl and these grids u have built and turned into diff worlds help us for get most of our pain and limitations…in inworldz and sl help us heal our selves and meet ppl from all over the world here in our computers. keep up the good work …i wish i would hace learned more about computers when the timex sinclair 2000 came out in the early 80′s
Scale tech | J2cab says:
September 16, 2011 at 11:28 am (UTC 0)
Reply
[...] Internet scale architecture and InWorldz » InWorldz Tech Blog [...]
Dead-Cat-Bounce and other stuff… | soror Nishi says:
September 23, 2011 at 9:11 am (UTC 0)
Reply
[...] impressive thing I have seen this week is a post by Tranq of InWorldz fame who posted details of their server layout in ground-breaking (for me) post explaining stuff that I mostly didn’t understand. Both Wizzy [...]
no bullsh!t at InWorldz at i live in science land says:
September 28, 2011 at 9:49 pm (UTC 0)
Reply
[...] their post on their tech blog, they outline their infrastructure at Cari.net (Cari and LimeStone Networks [...]
Internet scale architecture and InWorldz » InWorldz Tech Blog « Marina Sport Racer Sailing Club says:
November 30, 2011 at 11:47 am (UTC 0)
Reply
[...] Internet scale architecture and InWorldz [...]
Response: InWorldz at three – Hypergrid Business says:
April 8, 2012 at 1:15 am (UTC 0)
Reply
[...] laid out our outward scaling model, for those who are so inclined to know more, you can read our TechBlog and David does an excellent job laying out what we’ve [...]