Categories
Blogs News Tech Blog User Blogs

ROBStock 2015 Music Festival

ROBstock is coming to 3RG very soon 27th -29th March and for a very worthy charity ‘Doctors Without Borders’ .This woman here is missing 2 acts on Friday, so if anybody’s up for it- musicians poets ,storytellers get in touch with me . You don’t wan’t to hear me sing … I can clear a shopping mall just humming! So look at the website and give the grid a break …
Check here for vacancies and information.
http://robstockfestival.org/banner for website

Can’t come ,maybe not a member of 3RG and don’t have geodes, but want to make a donation? Well you won’t miss out . We are streaming live to Twitch TV and 3RG Radio all weekend.

This is where to go , and you can donate in any currency, a large amount or small. Its all going to the Doctors, plus we have geode tip jars and links to this page that we have spread about the grid. So come and have a good time in a real festival setting – no mud- from the comfort of your armchair. And the price of a ticket? Well its not £130 a day – but a donation to this amazing charity would be welcome. Target $1000 … lets smash it.  https://www.justgiving.com/RobStock-2015

A big thanks to all the artists who have signed on to give their time .. and thanks to Zinnia Frenzy for getting them all to agree to share the talent and time for this great event.

Categories
News Tech Blog

UPGRADE TIME!!!

On Friday November 21st at approx. 7am EST we will take the grid offline to perform an upgrade.
This upgrade will include moving us to the latest code and will also bring HyperGrid to 3RG.
By default, each “OWNED” region will “NOT” be HyperGrid Enabled.
We will provide tools for our users to allow each to selectively choose which of their regions, if any, will allow hyper grid visitors.
Initially, our HyperGrid export permissions will be very limited as we want to be sure we don’t get hurt by this.
Our initial testing on the beta grid has thus far been very successful and we feel confident our upgrade will go well.

There will be many changes to our platform during this upgrade so we expect the “Main” grid to be down until approx. 8pm on Sunday evening (11-23-14).
During this time, our Beta Grid will remain online.

Thank you for your patience.

Categories
Tech Blog

Patch Applied….

Yesterday, November 11th, 2014 the developers of OpenSim released a security patch which has been applied to all regions of our grid.

Information about this issue as released by the developers….

“There is only one change in these releases. This is to fix an issue where llRemoteLoadScriptPin() does not treat the pin ‘0’ as an unset pin. By default, all prims have a pin set to 0. Therefore, this bug allows llRemoteLoadScriptPin() to specify a 0 pin to load scripts into owned prims with no pin set where this should not be possible.

Unless you are very sure that no user will run a script from an untrusted source, we would advise you to update as soon as possible. There are no database migrations or config changes in this release compared to the previous in the series, so all config files can be used without alteration.

This bug was introduced more than 6 years ago with the original llRemoteLoadScriptPin() implementation and so affects all versions of OpenSimulator at least from 0.4. If you are using a version of OpenSimulator older than 0.7.4 (which was released in August 2012) then you will need to upgrade or apply the patch in commit 5aa8ba1 manually.

Many thanks to Tranquility Dexter of Inworldz for pointing out the bug and the fix. “

Categories
Tech Blog

Our Backup/Recovery Plan

Due to the recent issues OsGrid had with their hard drive array, I have received many inquiries about how we would handle such a situation.
Backup procedures/restorations vary from person to person, while there are many “Common” solutions, there will always be an argument about which is best.
It’s like Ford is better than Chevrolet, Or Chevrolet is better than Dodge, Coke is better than Pepsi, etc.

 

By what I’ve heard, it seems osgrid had a raid 10 array which they used for their assets.
Raid 10 is typically a very good solution as not only do you get “Raid”, but you also get striping.
Striping is when data is stored on multiple drives instead of a single drive.
To be more clear, if I were to store my name; “Butch Arnold” on a single drive with no striping, my name would exist exactly as “Butch Arnold” on that single drive.
If I stored my name on multiple drives using striping, I might get the name “Butch” stored on the 1st drive, while my last name might be stored on a 2nd drive.
To some, striping is a common practice and when it is working correctly, helps to retrieve the data much faster because it can read the data from 2 drives (or more) at the same time.
Those who think striping is bad usually think this because troubles similar to those experienced by osgrid has happened.

I’m not a striping or raid “Hater”, but I’ve chosen not to use them due to this risk.

 

Instead, we have a primary database on one machine which feeds 2 other slave databases located on 2 more separate machines.
Each evening, we make a backup of one of these databases and we store it on that specific machine and we send a copy of that same backup to yet another machine and also to an amazon hosted storage area.

We also generate individual region oar files and region database backups each night and store them in a similar fashion.

I’m not saying this is the way it should be done, but this has worked well for us.
It has allowed us to retrieve a backup of a specific region using either a database backup, or an oar file whenever we’ve needed it.

 

All data storage methods will fail at some point, you can count on that.
The trick is, in my opinion, to be ready to restore your data using backups stored in multiple locations as it is unlikely that all of these separate storage devices will fail at one time.

I’m not second guessing anyone here, the operators of osgrid I’m sure had systems in place, and it’s unfortunate that they failed.

In our configuration, if our main database fails we can quickly switch over to one of our slave databases.
If, for some reason, our slave databases contain corrupted data and cannot be used, we can then install a completely new database and rebuild it using one of our backups to restore our grid to that point.

 

My thoughts on backups are to do them regularly, store them on several physical machines, and have a plan in place to restore if needed.
Even our backup plans could fail due to unforeseen circumstances, but it is highly unlikely that our main database would fail, and our second and third databases would fail, and even more unlikely that each one of our backups are lost since they are stored on separate machines.

What happened to OSgrid could happen to anyone, the lesson to be learned here is to be ready in case it happens to you.

 

Another part of this which helps me sleep better is the fact we own all of our equipment.
We have 2 “Spare”, “HOT” servers running online and are there if we need them in a hurry for grid use.
We also have a 3rd spare server which is offline, but can be placed online in an hour or so and be ready for our use if needed.
We use both HP and Dell servers and we have the in house knowledge and ability to do repairs and upgrades on them as needed.
The datacenter we use is located just a short drive from me and I can go there at ant time, day or night and work on our servers myself.

While our solution may not work for some, it has worked well for us thus far.

Categories
Viewers

Firestorm Voice Issue……

In the past we’ve asked our community to disable voice if they were using the Firestorm viewer on 3rd Rock Grid.
Recently, We have made some changes to our configuration to lessen the issues when using Firestorm with voice on our grid. Please continue to use the Firestorm viewer with voiced turned on, unless there is a problem.
So far, the early results look very good but we encourage our users to provide feedback of any and all issues while using the Firestorm viewer.

Categories
Tech Blog

Server issue overnight (USA TIME)

We experienced an issue overnight with one of our servers which caused many regions to become unresponsive. We were able to isolate the issue, correct it and restart all regions on that server.
Those regions are back online and responding normally.

Categories
Tech Blog

Add New Events Tool Available….

We have now added the ability for our users to add his/her own events to our events calendar.
Simply go to “World” on the main menu of the website and hover over “Event Calendar”… a new sub menu will appear.. “Add Grid Event”. Click “Add Grid Event”, enter the required information and click the button “Post Event”. Your new event will be immediately available via the website Events Calendar and also in the Grid Viewer under search… events tab.

Categories
Tech Blog

Sit Positions after upgrade…..

During our previous upgrade on Dec. 26th, 2014 we decided to move back to the “Out of the box” sit positions.
Several years ago the “sit” position in OpenSim was different than it is today.
When the change was first made, we decided to stay with the old sit position which caused us to change each new version before we upgraded to maintain this old sit position.
This was great for our current users at the time as they would not need to change the sit positions in all of their items, but.. it has always caused issues for new users coming to our grid from others since many of their scripts, etc would be using current sit positions while we were still using the old ones.
The decision to change back to the “Stock” opensim sit positions was not welcomed by many, but this was a needed change to make sure our grid is inline with what new users coming to our grid expect. We do not want to provide any reason for a new user to not want to be a member of our grid. This was an issue for many new users and we thought it best to make the switch now, as with each new user increase, those who would not be happy with the changes later would increase.

Categories
Tech Blog

Troubleshooting Region Lag

One of the common questions and complaints is concerning region lag.
There are many issues which can contribute to “Apparent” region lag:
1. Your internet connection speed
2. Your computer hardware – CPU, Ram, and Video Card
3. Number of people in your region
4. Number and type of prims in your region and surrounding regions
5. Textures in Your Region and surrounding regions
6. Attachments on yours and other avatars
7. Scripts

The most obvious, but commonly overlooked issue is your connection speed. The region needs to send and receive data to/from your viewer, if your connection speed is slow, this can give “You” the appearance that the region is “Laggy”.
Sure, you may pay big bucks for a fast connection, but it may not always be fast and should always be the first thing you check. You can also have someone else come to the region and see if they have the same experience, if they do not, it is likely not the region.
Remember, “Your” connection to a region is “NOT” the same as another user… unless you are both using the same internet connection. Each user’s connection follows different paths to communicate to the region servers and “Any” trouble along the way can cause problems.

I don’t have any issues on “XYZ” grid… why do I have them here?
Again, your connection to “XYZ” grid may be shorter, better, etc., unless they use the same provider as us, your connection variables will be different with them.

When it comes to computer hardware, there is no general rule of thumb, but you should expect better performance from newer, better technology. Your graphics card is probably the most important aspect of your hardware as it will see the most load.

Your region will generally perform better with less people in the region. As the number of users increase, so will the demand on “All” variables. Your hardware will need to work harder, our hardware will need to work harder as more data is moved around.

The number of prims in a region will affect performance of a region. Each prim has an impact on the physics engine of the region as will each texture. The size and type of a prim matters as well… a torus will impact the region more than a standard cube. The type and size of textures is important.
The larger a texture is, the more time it will take to download/process and display.

If your draw distance is set high enough to see into a neighboring region you will need to account for the downloading and displaying of all prims and textures in that region as well, including any attachments an avatar there might have.

Scripts really don’t have as much impact on the regions performance as some might think as other things take priority and what is left is used for scripts. This is “Generally” the case.

So how can you troubleshoot a laggy region?
You can start by using some of the viewer’s built in tools.
There will be a “Statistics” option in your viewer… find it, open it and make some observations.
StatisticsBar
The FPS and Bandwidth readings at the top under the basic heading is “Your” FPS and Bandwidth stats, to see the region’s stats you’ll need to look lower under the simulator heading.  Simulator FPS values of less than 43-44 will start to display “Chopped” movement.
The time dilation is a very important value. 1 is a very good and represents the region is seeing time in real time. A general rule would be that a region with a dilation reading better than .89 is not lagged at all, while a region with a reading of .1 is ready to crash.
Look at the Total Frame Time and all sub categories.. you will see a spare time category which represents how much free time a simulator has to perform other work. If this reading is very low it means your region is loaded and doesn’t have any “rest” time.

Best use policies:
1. Use the smallest texture possible. A small texture will require less data transfer and will load your video card less.
2. Position your prims to cause the least amount of collisions.
3. Set your draw distance to the lowest setting that is still acceptable.
4. Try to use as few complicated prims that you can.. a torus will require much more overhead than a standard cube.
5. Change your graphics settings in the viewer. You can find a fair amount of performance gain simply by tweaking some of these settings and in many cases, you won’t “See” a difference.

Categories
Tech Blog

Upgrade Summary

On December 26th, 2013 3RG was taken offline to upgrade our version of OpenSim.
3rd Rock Grid is now running on version 7.6-Release code, but with a few custom changes.
During this upgrade many other changes were made to our platform to increase our stability, performance and reliability.
This upgrade combined with our other changes have provided a very nice platform on which we will continue to improve.