2018-05-11 Blueprint Validation Changes

Details about updates to Shores of Hazeron

2018-05-11 Blueprint Validation Changes

Postby Haxus » Fri May 11, 2018 2:54 pm

Blueprint Validation Changes
Servers are running out of memory.

The problem was traced to a small number of troublesome blueprints. Nine of them uncompressed to over 1GB of memory each. The servers have to load and uncompress designs to use them. Some of the older servers only have 8GB of memory.

  • Developed tools to analyze and report on all designs in the database, to find problems.
  • Tightened restrictions on design and assembly blueprint validation.
    • Made the compressed size limit on spacecraft blueprints a hard ceiling. There is no longer a cushion between the reported maximum and the actual maximum.
    • Imposed a limit on the uncompressed size of blueprints. This is in addition to the previous limit on the compressed size of blueprints.
    • Changed blueprint validation reports to show the amount of memory consumed by the compressed and uncompressed blueprint, in bytes.
    • Changed blueprint validation reports to show the compressed and uncompressed size limits, in bytes.
    • Added a check for duplicate parts. A design is not valid if any two of its parts are exactly identical. There were several designs with thousands of the same identical part.
    • Imposed a limit on the number of guard posts in a design.
    • Imposed a limit on the number of citizen posts in a design.
    • Imposed a limit on the number of livestock posts in a design. At least one design had over 2000 livestock posts; several others were close to 500.
  • Built new client programs with these changes
  • Built new server programs with these changes.
  • Shutdown servers.
  • Made a backup of the database.
  • Analyzed all current building and spacecraft blueprints for validity.
    • Out of 784 building designs, 51 were found to be invalid.
    • 474 buildings referencing bad designs were deleted.
    • 35 building blueprint exchange records referencing bad designs were deleted.
    • Out of 1589 spacecraft designs, 520 were found to be invalid.
    • 243 spacecraft referencing bad designs were deleted.
    • 88 spacecraft blueprint exchange records referencing bad designs were deleted.
  • Created a detailed report of all invalid blueprints, the same report produced when they get finalized.
  • Created a .SoH file from each invalid blueprint, in case there were no other backups. If you want your deleted designs, just ask.
  • Deleted all invalid blueprints from the database.
  • Deleted all buildings and spacecraft from the database that referenced invalid blueprints.
  • Deleted all design exchange records from the database that referenced invalid blueprints.
  • Restarted servers. - - -Currently at this stage of the process- - -
  • Rescued people stuck in limbo as a result of their ship or building being deleted.
Let me know if you would like items or cargo recovered from a ship that was lost in this process. I will need its name or, if you had a berth aboard it, I can find it from that using the backup; you don't have to keep your berth for me to rescue the items or cargo aboard your ship. The name of an officer or crew aboard it would also be enough to find it.

This process took much longer than expected. I credited all active accounts with 3 days of time, to compensate for the down time. Thanks for playing.
User avatar
Haxus
Site Admin
 
Posts: 3030
Joined: Fri Jan 14, 2011 8:00 pm

Re: 2018-05-11 Blueprint Validation Changes

Postby Deantwo » Fri May 11, 2018 3:18 pm

Haxus wrote:Added a check for duplicate parts. A design is not valid if any two of its parts are exactly identical. There were several designs with thousands of the same identical part.

Bye bye TARDIS designs. *waves*
You won't be missed.
AnrDaemon is the solution to the [s]Fermi Paradox[/s] Hazeron suggestion flood problem, the great suggestion filter.
User avatar
Deantwo
 
Posts: 5162
Joined: Fri Jan 25, 2013 4:38 am
Location: Rævehale

Re: 2018-05-11 Blueprint Validation Changes

Postby Vectorus » Fri May 11, 2018 3:35 pm

I think every ship currently keeping my empire going will fail the size criteria - is that the case? Since the limits were imposed and the warnings about possible deletion were posted, I've carefully avoided modelling any ship from scratch which exceeds them, but most of my infrastructure dates from before then.

I'm not sure I could bring myself to start again, if they all disappeared. That could be it for me, at least for a while. I'm glad you found the problem, of course, for everyone's sake! The servers are what they are.
User avatar
Vectorus
 
Posts: 184
Joined: Sat Mar 03, 2012 10:21 am

Re: 2018-05-11 Blueprint Validation Changes

Postby Onepercent » Fri May 11, 2018 3:37 pm

Haxus wrote:Blueprint Validation Changes
Servers are running out of memory.

[*]Added a check for duplicate parts. A design is not valid if any two of its parts are exactly identical. There were several designs with thousands of the same identical part.


Does this mean identical in position as well or just two identical parts that are in different locations? if so; how will having same size rooms work?
User avatar
Onepercent
 
Posts: 192
Joined: Thu Mar 10, 2016 3:23 am

Re: 2018-05-11 Blueprint Validation Changes

Postby Deantwo » Fri May 11, 2018 3:41 pm

Onepercent wrote:
Haxus wrote:Blueprint Validation Changes
Servers are running out of memory.

[*]Added a check for duplicate parts. A design is not valid if any two of its parts are exactly identical. There were several designs with thousands of the same identical part.

Does this mean identical in position as well or just two identical parts that are in different locations? if so; how will having same size rooms work?

Pretty sure he means identical location too. That is how the "TARDIS exploit" worked, by stacking the same hull parts on top of each other to increase volume without increasing the size.
AnrDaemon is the solution to the [s]Fermi Paradox[/s] Hazeron suggestion flood problem, the great suggestion filter.
User avatar
Deantwo
 
Posts: 5162
Joined: Fri Jan 25, 2013 4:38 am
Location: Rævehale

Re: 2018-05-11 Blueprint Validation Changes

Postby Vectorus » Fri May 11, 2018 3:47 pm

A possible exception should be made for field sites. There's really no reasonable way to get a degree of food production anywhere close to the shop and house numbers of new buildings, otherwise. You're looking at 4 or 5 fields per building compared to 65,000 residents or worker processes. You would need to use vast numbers of buildings on farms, even if you were only using 100-process/house buildings. Perhaps food production should just be tied to shop volume, instead.

Even some sort of grace period would be hugely appreciated. Back before the limits were introduced, I poured a lot into making ships worthy of the new designer. I wanted to embrace the change, so I really rely on those now. Probably 100% of my ships are in trouble; 90% of my empire population is in farms with copied fields, because I thought they would need enough food, then used the same design as a valid house too, once I realized they didn't.
Last edited by Vectorus on Fri May 11, 2018 3:52 pm, edited 1 time in total.
User avatar
Vectorus
 
Posts: 184
Joined: Sat Mar 03, 2012 10:21 am

Re: 2018-05-11 Blueprint Validation Changes

Postby Haxus » Fri May 11, 2018 3:51 pm

It was time for hard choices.

Before going to bed last night, I decided to abandon this project altogether. I was out of options and had yet to find the cause for the massive memory consumption on the servers. This project is a constant drain on me financially. It is impossible to justify the enormous expense required to add lots more memory to the servers.

Overall, a very small number of designs were affected, a tiny fraction of the designs that have been made. I will add the numbers to the first post as I get them. Most of the invalid designs were made by only two people. I studied the effect on existing designs to minimize the number that became invalid.

Infinite detail is a fun concept. It is not compatible with finite servers.

I sincerely apologize if this change drives anyone away. It was you or me.
User avatar
Haxus
Site Admin
 
Posts: 3030
Joined: Fri Jan 14, 2011 8:00 pm

Re: 2018-05-11 Blueprint Validation Changes

Postby Haxus » Fri May 11, 2018 3:55 pm

Does this mean identical in position as well

Yes, identical in every way. I can think of no situation where this should be valid.
User avatar
Haxus
Site Admin
 
Posts: 3030
Joined: Fri Jan 14, 2011 8:00 pm

Re: 2018-05-11 Blueprint Validation Changes

Postby Vectorus » Fri May 11, 2018 3:59 pm

^Just edited the post before yours.

Of course, I'll wait and see what happens. You need to do what you need to do. If I feel a bit shell-shocked by suddenly losing all my work, that's only my fault for investing so much. I might need to take a break from actively developing things and playing, but you wouldn't have driven me away. I would keep my subscription going and I'd still stop by to chat, no worries! It's just...there may be other projects which it is safer to pour that amount of research and creative effort into, like my book.

I'm almost certain there are more than nine Dulcinee and Falbala subtypes, so perhaps those have escaped...but then, only a few of them have actually been manufactured. So don't know.

Again, a grace period would soften the blow for the few of us concerned. I could at least find alternative homes for most of my population and replace my continuous missions with some stopgap designs. But if it's costing you money every moment, of course that may not be possible.

Thanks for all the work, my friend!
Last edited by Vectorus on Fri May 11, 2018 4:56 pm, edited 2 times in total.
Image
User avatar
Vectorus
 
Posts: 184
Joined: Sat Mar 03, 2012 10:21 am

Re: 2018-05-11 Blueprint Validation Changes

Postby Haxus » Fri May 11, 2018 4:09 pm

Here is the most extreme example.

A design was composed of a hull box that is the maximum dimensional size allowed. It was made transparent so it required blending, which is slower to render. That box was repeated on top of itself almost 5000 times, until the compressed size of the design was at 98% of the compressed size limit.

Uncompressed, it was over 1GB in size.

Adding insult to injury, the architect of that design had the nerve to complain that it caused lag.

That same architect made a farm with over 2000 livestock posts exactly on top of each other. If I had spawned a cow at every one of them, would they have complained about the lag?

This is what I am up against.
User avatar
Haxus
Site Admin
 
Posts: 3030
Joined: Fri Jan 14, 2011 8:00 pm

Next

Return to Updates

Who is online

Users browsing this forum: No registered users and 2 guests