Fixing the problems of Second Life: a hill too high to climb?

The platform design has been showing its limitations since 2006 and  I believe a lot of the platform issues are directly related to the system design.  From concurrency issues to the basic user tools, most things are hampered by the design.  This isn’t going to change as the time for a redesign has passed and I do not believe there is the desire nor the money to undertake it.  So we end up with a sub-optimal platform held together by various fixes that may or may not work.  I stopped bringing people into SL back in 2008 as I found that virtually everything I had to teach them to do required a work around or an explanation of what should happen against what actually happens. It directly impacted on their ease of assimilation into this world of ours – amongst other things.

So, here we are at the latest birthday celebrations for SL.  Officially 7 years old and still with us along with the flaws.  Every year since 2007 the SL birthday celebrations have suffered from the same thing – lag and the inability of the regions to hold more than what is really a token number of people.  Philip Linden stepped out in public and made a speech there which is what triggered this post.

Phil, in his introduction to his speech at SL7B said the following:

Philip Linden: Okay everyone! I realize not that everyone there can hear me and I’m sorry for that, but I’ve got until 11:30 this morning and then I’ve got a drop-off at summer school to do, so I absolutely have to leave. So I just wanted to get started and again, I’m sure somebody will — maybe if we’re lucky here — will do the favor of recording and translating me. I can’t type as quickly as I can talk, so I’m not going to try to.

Two years in and they still don’t have a stable voice system and they persist in using it despite the fact it is sub optimal – just like they insist on using voice for resident meetings when they know that not everyone uses it, it doesn’t work well and there’ll be no readily available transcript. The cavalier attitude towards the userbase can be summed up in this speech.  Despite the problems of the last week and the lack of confidence in the future of the platform, the Chairman of the Board didn’t spend a lot of time on thinking about what he wanted to say to his customers.  My personal preference is for text inworld and in this case a carefully worded speech designed to boost confidence should have been a must – copying and pasting it in would have been a lot quicker and Phil would have ensured that everyone could see it and it would be verbatim, rather than a transcription that may be prone to errors.  Voice has failed senior management on so many public occasions yet somehow they don’t see it as a priority to fix – unless of course the platform is so hampered that there is no fix.

He then went on to laud the achievements of the SL platform:

You know, you might jump up and say, “Hey Philip, of course there’s so many things you could’ve easily done differently that in these last 10 years that would’ve made things better — or executed better — but you know, changing history has the risk that you might have done something that broke everything in some way, and I wouldn’t toy with that. I think what we’ve achieved here is a magnificent accomplishment together — all of us, the Lindens, the Residents, the Lindens that aren’t with us anymore — we’ve all worked together to build something just incredible. And I wouldn’t even take any chance at anything that might mess it up, it’s unbelievable what we’ve achieved.

I was thinking about this — what to say today and what to talk about — and I had a thought. I wanna try something. I wanna read you guys just a quick list that I made this morning, so bear with me and let me read you a list of stuff here:

Our financial fraud detection systems; the systems we use to transfer assets from the Teen Grid; the central databases; our dark fiber backbone; our asset servers which have about 450 terabytes of data; the 40,000 simulator cores in the system; the group chat system; the LindeX Market placement and fulfillment systems; the physics core; the visual rendering system; the scripting engines; the ability to transfer and move land; the region conductor that manages all the sims coming online; the map servers; the inventory servers; the client UI; the content takedown tools; the monetary policy, processes and systems we use; the customer support tools; the Department of Public Works; our international payment systems; our backup systems; Linden Homes; the Welcome Islands; the Infohubs; the grid monitoring tools; the localization systems; the private regions; our land auction systems; forums; search appliances; the Support Portal; metrics dashboards; our Phoenix, Dallas, and Washington, D.C. data centers; our 3rd-party Viewer directory, open source repositories and programs, and our internal build systems. 

From a customer perspective my response to this list of achievements is this: The asset servers constantly have problems and customers still regularly lose or have items corrupted, group chat has suffered for years from overload and lag, mono has created more problems than intended that haven’t been resolved yet, the new client UI has been a flop, the content takedown tools may be nice but the human interaction needed to use them isn’t working that well as stolen content is left inworld long after being reported and in breach of the (as I understand it) 24 hour takedown timeframe specified for DMCAs, the search appliance is so defective it’s not fit for purpose, the support portal may now look glossy but the support staff can’t resolve most problems and make you file a ticket that is then ignored for weeks at a time.  

Tell me again how these have improved the user experience?  The LL retort will probably be that the last year was spent on stabilising the infrastructure. Sounds good until you ask yourself what everyone else in the company was doing.

On top of this the platform is showing its age.  I don’t know if the heralded mesh import functionality is going to make it look more sleek but it’s a step in the right direction for visual appeal.  What we don’t know is how it will impact on the rest of the platform and the general usability.  Sculpties allowed for more natural builds but they come at a price – viewer lag. What overhead mesh will bring is still to be determined.

Yet it doesn’t fix the problems with the infrastructure.  Group numbers are too limited for a social platform and the design means it can’t be easily solved and there’s the issues around lack of functionality to manage the groups and roles, TPing is still an issue, the items lost is an issue, voice is problematical, the issues around the search appliance can’t be easily fixed because of the way the data relating to regions and picks and classifieds are stored – which has been causing problems since at least 2007, let alone trying to modify the appliance to use the data structures used in SL.

On top of that, quick wins like incorporating some of the third-party viewer functions (double-click TP, doubling the attachment points, temp uploads of textures, previewing animations on the avatar, easier searching in the inventory, the inventory worn items tab, built-in radar and AO) have not been adopted.

Then there’s the question of whose user experience they’re trying to improve which brings us back to the old discussions of augmentalists versus immersionists.  Will Linden Lab ever admit that these are two distinct groups and forcing the immersionists to change won’t work and the augmentalists won’t stay as it is currently?  I think there’s room for both but it will take careful management of the delivery of the tools to attract the augmentalists without impacting on the immersionists – who are the people keeping them financially afloat. They’ve not demonstrated the subtlety of thought to give me confidence they could achieve it.

One thing is certain, the delays between identifying a problem and providing a workable solution has to decrease.  4 years since user retention and the UI became an issue it still hasn’t been resolved and it was only this year that the initial concrete steps were released. Windlight was never fully deployed, mono is still an issue.  At the snail’s pace LL responds to issues that directly affect the experience I fully expect them to still be issues when SL finally closes its doors, as I don’t see the *cough* restructure optimising the delivery of the fixes.  It may have focussed the minds of those who are left but the organisational management systems are either non-existent or so defective that improving the efficiency and effectiveness of delivery won’t be an outcome.   I’ll add this link for those of you who need proof that LL are just not good at things like prioritisation or urgency  Ever wonder really about the ‘most hated bugs’ on the Second Life JIRA and how they are doing? | KessKreations.  

So is it one hill too many?  While the platform is the major hurdle to delivering new and exicting content and features I think so.  That’s not to say that it can’t be optimised to at least improve retention rates for existing and new users and stimulate activity but the glory days are over.

And Phil, we haven’t built this together – LL has openly despised its user base and told us to move aside.  Linden Lab may have provided the platform and then made it as unstable as it could through poor business practices but despite you we’ve built this world. LL may have tried to cash in on our imagination, creativity and time but what is here is still ours – we’ve made it and we can take it back and as you can see that’s just what we’re doing.  Don’t go blaming us for your ineptitude by claiming your failure is a joint effort.

Before you shrug and move on gentle reader, here’s a few quotes from two educators blogs and how their students react to SL.  These should be the natural customer base since they’ve been brought up on computer games and are computer literate – yet as you’ll read they’re rejecting SL as an entertainment platform of choice.  I don’t think their reasons are much more different to those of the other 93% of people who have tried Second Life and abandoned it.


MixedRealities :: Virtual worlds crisis develops into virtual currency upheaval

Second Life is more than just another social network. It is an immersive space and the users (actually ‘residents’) are emotionally attached to that world. As I wrote in some other post, the depth of that engagement and the serendipity of the encounters are both a strong and a weak point.

A strong point, because who would be against depth of engagement or against the many possibilities of meeting fascinating people from all over the planet?
A weak point however, because becoming a resident is like becoming an immigrant: you partially leave the people you know and you start an adventure, discovering new people, new venues and even new jobs. That sounds great, but for many people it is more than they wish to handle, and they prefer to stick to the people they do know already, using networks such as Facebook to enhance contacts they basically already have.
That’s a structural dimension which cannot be solved by yet another viewer. Maybe an integration in browsers or in social networks such as Facebook would help, but the fact remains that engaging yourself in an immersive user-generated world will probably change your life, and that’s not what most people are waiting for.
This does not mean that there is no future for user-generated open-ended virtual worlds. As long as you have a loyal and intensely creative community, the project can go on. New platforms such as browsers, mobile devices and tablets can help to bring in more people in the virtual worlds space. Let’s not forget that a lot of what is happening is based on what the users create themselves, and open source developments such as OpenSim become even more strategically important.

and from the comments


You state something I see every time I teach a class with SL, “the fact remains that engaging yourself in an immersive user-generated world will probably change your life, and that’s not what most people are waiting for.” My students end up enjoying the experience, but they don’t return to SL after the class ends. They prefer augmented experiences, not immersive ones.

Has the Lab ever considered what the human cost would be if some of the renters I describe–often with virtual partners, romances, etc.–lose their sandbox? I’m an educator and can build simulations elsewhere.

It’s so easy to make fun of SL‘s most dedicated fans, but I worry about them if LL shuts its doors.


What?s going on at Linden Lab? Virtual Education

What is it that we, especially those in education, have been asking for years to be a priority? Ease of use . . . this is the holy grail of the SL platform. Here we are seven years down the road from the official launch of the platform and ease of use is still an issue. The controls for avatar navigation, chat, and content creation have changed very little over the years. As a matter of fact, with the numbers of students I’ve run through SL over the past four years, the most frequent comments are usually on how “clumsy” the controls are, how “dated” the look is, how “difficult” everything is to do, and how “slow” everything is in SL. And one of the biggest complaints out of the teachers has been how “unreliable” the platform is when they’ve planned a lesson in SL (maintenance and downtime). It seems that the teachers are more enamored with SL than their own students are.


Second Life: The beginning of the end?

I haven’t posted much this year because for a lot of it all I have seen is failure and there’s enough people out there who have been pointing this out without me adding to it.  The announcement of the redundancy of 30% of the staff at Linden Lab started me reviewing the last 4 years of watching Linden Lab as they’ve thrashed around.  How did a platform that really did have the potential to be a web changer end up like this?  This restructure is the last opportunity for Linden Lab to get Second Life at a stage where it is sustainable for the longer term and at the moment I see the chance of success as problematic.    

All I’ve seen down the years is opportunities wasted and poorly thought out and executed policies. Looking at it objectively they took the mantra of low quality is good enough for the web down to new levels. This, combined with their take it or leave it approach to their customers, has done more harm than good. Linden Lab has never really made an effort to stop treating its customers with disdain – from the April fools MotDs telling us to get a life back in 2007 to M telling us to move aside in 2009 and the general ignoring of customer concerns it’s all added to reducing the goodwill of the customer base.  M wanted us to move aside and for Linden Lab the problem is we have moved aside – so far aside in fact that a lot of us are gone and now Linden Lab is suffering.

So, off the top of my head and just in case Linden Lab ever want to do a Lessons Learnt exercise, here’s a list of some of the more obvious failures. They’re in no particular order as when I looked at them it was apparent the inefficiencies are spread throughout the entire organisation.  It’s not just policy but development, customer relations, customer service, financial controls, quality control – it’s a systemic organisational failure we’re looking at here.

So, here’s my list:

  • new user experience.  In 2006, user retention was an issue.  It took until 2010 for action to be taken.  There was no real information for new users, no support during the learning curve and no protection from the troublemakers who took great delight in ruining the initial experience.
  • Opensourcing of the viewer. The open sourcing of the viewer was successful as it’s removed some of the pressure on Linden Lab to deliver a viewer that suits all user requirements. (note: this has been changed from the original – see comments below)
  • Script Load:  In 2008 script numbers were beginning to become a problem, March 2009 the first monitoring tool released on platform,  June 2010 – still nothing concrete done.  
  • Viewer v2:  Years too late to catch the wave of new users and obviously unsatisfactory to all but those currently inworld who do more than use it as a social platform. Had Linden Lab actually told people that this viewer was solely aimed at new users they wouldn’t have had so many howls of outrage at the lack of usability.  
  • GSA Search: Deployed in prototype without any apparent understanding of the limitations of the current Second Life infrastructure it was going to be run against nor any great understanding of the data used by GSA. Two months after the last major release of the new search it still has quite serious defects that have not been addressed.  As usual Linden Lab continues to take money from businesses inworld without providing the service paid for and without any acknowledgement or compensation.
  • Removal of images on Xstreet SL and the blog:  They were just unilaterally removed.  Better option would have been to allow images from certain sites while barring the rest.  It effected my Xstreet listing layouts and it’s quite apparent that something is missing from the ads but considering I spent weeks updating them all last September when they first requested we clean up the listings and insert images, I had no intention of doing it again for what was supposed to be a short time frame between the image removal functionality and the deployment of the new site..  Except now we don’t even have a deployment date for that.
  • Despite the malware that was embedded in some websites when the web tab was first released they never took any action to resolve that and with viewer 2.0 web-on-a-prim the same issues can occur and again Linden Lab has not put in any warnings about this and has left the default switched to on. 
  • Viewer memory leak: Fixed early 2009, something like 2 years after it became an issue. I saw a reduction of 60% on usage, which is a huge number, particularly for those countries that have to pay by usage,
  • Traffic bots: Another policy change that took more campaigning efforts by customers than it should have and when implemented was not enforced. They are still being reported and Linden Lab doesn’t take action.
  • Linden Homes: why increase the cost of servicing a premium account for current users?  There were better ways of deploying this idea that would have been more cost effective.
  • Windlight: initial deployment only and feature list never fully deployed. 
  • Mono: initial deployment and no fix for the issues it generated. They’re still working on it by all accounts.
  • Opensims: pricing model incorrect and no load management put in place. 
  • Adult content: A good idea but not really implemented past a policy posting and moving a few businesses to the lands.  Transgressors still being reported and ignored, Adult content providers being charged the same as other landowners but with less features.  Which the land owners were never made aware of during the migration.  All it really did was disadvantage those who supported and complied with the policy, while rewarding those who didn’t.
  • Ad farming: Another policy change that took more campaigning efforts by customers than it should have and when implemented was not enforced.
  • SL blog: released as beta but more of a prototype and no real changes were made in response to user feedback.
  • Xstreet SL login page:  has instructions to web dev still on it!! By the look of it no one ever bothered to check it prior to release and no one has looked at it since.  The quality assurance is indicative of the rest of the LL deployments.
  • Enterprise platforms: Linden Lab have never demonstrated value to business of using Second Life. It would have worked in 2006-7 when there was still apparent potential but by 2009 it was far too late.
  • Event listing issues:  never fixed nor cleaned up to remove the worst transgressors which made it difficult to easily extract information.
  • They have no demonstrable financial controls, not even a customer billing application.
  • Architecture was showing flaws in 2006 – it needed a redesign then.  It’s not going to happen now as it’s too late to justify the cost.
  • Disregard of their own policies when it suits their short term goals (and we’ve seen how successful that hasn’t been in the past) allowing  people who they know are under 18 on the main grid, allowing people who have had accounts previously banned for antisocial behaviour, allowing certain accounts to flagrantly flout policy when Linden Lab have been made aware of it.
  • Taking action against accounts and not providing reasons why.  Doesn’t give the account holder a chance to modify the behaviour as they have no idea what they have done wrong.  So they just come back under another account and carry on.
  • Avatars United.  A great idea which would allow Linden Lab to own the virtual user identity version of Facebook.  Has languished for lack of corporate support and marketing. This is probably the only asset that Linden Lab has left that might be attractive to a buyer – apart from the customer data held.
  • Deliberately alienating the education sector.  That one beggars belief.
  • Lack of action on user to user exploitation: content theft, scams and now unauthorised data collection by third parties – some who shouldn’t even be on the grid anyway because they’ve previously been account banned.

and the list goes on..  

The consistent themes for failure are pretty apparent.  A lack of management control on timeframes for deliverables, a lack of understanding of key objectives, lack of financial controls, lack of acknowledgement of responsibility to customers to deliver something useable and an all round lack of attention to detail and an understanding of their product.  The organisational inefficiencies are pretty breathtaking, the sheer amateurism of Linden Lab as a business is astounding and despite the “restructure” I don’t see that changing any time soon.  There’s no indication of any financial or development discipline being implemented as part of this change so it’s business as usual.

Here are a  few best practice tips:

  • Developing and testing in production is a recipe for disaster,
  • Taking years to deliver something that affects your core business is unacceptable,
  • Ignoring/despising your current cash flow generators is also not a sensible business move, particularly in the current financial climate,
  • Allowing your staff to work on anything they feel like without a business case and a cost/benefit analysis is asking for trouble. In fact allowing any activity to go ahead without them and a RAID is undesirable,
  • You might like to also try some risk/impact analysis sometimes too.