Game development is fragile

By on February 3, 2013 6:28:29 PM from JoeUser Forums JoeUser Forums

Frogboy

Join Date 03/2001
+1089

Full priced Stardock title releases of the last 5 years as measured on Metacritic.

image

One of these is not like the others.

22 Replies
Search this post
Subscription Options


Reason for Karma (Optional)
Successfully updated karma reason!
February 3, 2013 6:45:39 PM from Elemental Forums Elemental Forums

Even if the anomaly is ignored, it's a downward trend overall.

 

Reason for Karma (Optional)
Successfully updated karma reason!
February 3, 2013 7:43:59 PM from Elemental Forums Elemental Forums

Can't expect every game to get 90+, all the scores but WoM are good.  

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 8:29:00 AM from Stardock Forums Stardock Forums

Quoting Lord Xia,
Can't expect every game to get 90+, all the scores but WoM are good.  

Sure, the individual scores are good, but the fact that they are trending downward over time should be of concern.

Is it because consumer expectaions are changing? Is it technical?, Is it staffing levels (workload)? Skill set distribution?, etc.

I'm not familar with the software industry, but these are the kind of things I would consider where I work.

Reason for Karma (Optional)
Successfully updated karma reason!
Sign Up or Login and this ad disappears!
There are many great features available to you once you register. Sign Up for a free account and browse the forums without ads.
February 4, 2013 9:36:18 AM from Elemental Forums Elemental Forums

I think Galciv2 Twilight being the highest rated game speaks volumes about the rating system. I am still waiting patiently for Stardock to implement fixes to allow the AI to use all the racial specific tech. How do you get off selling a single player game thats AI cant use the major feature advertised? Your top rated game was a scam.

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 9:37:16 AM from Elemental Forums Elemental Forums

I've got some experience with statistics (I had to take it in college and did mediocre in it).  This isn't a good graph to really show a "trend".  The scale should go from 0 to 100.  Also, there are too few scores.  Also, reality check.  Getting a game review in the 90+ is exceptionally rare, and you simply can't consider that the standard to which you should compare all other released games.  First game is a 95, all other games should be 90+.  Prepare for disappointment.  Should be proud of anything in the green, 76+.     

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 9:43:10 AM from Elemental Forums Elemental Forums

Actually, I'm not so sure that the trend necessarily indicates Stardock is doing worse.  You have tried a number of genres, and you've had to learn some new things along the way.  Not sticking to a formulaic franchise means that the games you come up with are fresh.  What you also need to factor in is the post-release support of those games which makes those games better by spades by listening to the players.

You need to consider that those scores are averages (right?)  and therefore there could be a lot of variation in there, so what you would really want is to ask Metacritic nicely to send you a scatter plot for your games, unless you know a way to scour that data yourself.  A game that has three reviews with high marks and two with low marks obviously gets a lower average than one with five high mark reviews, but unless you know how high and how low, it's not much to go on.

Personally I would rate Fallen Enchantress a bit higher than either Dark Avatar or Twilight of the Arnor, and I played a lot of hours of those.  The reason I say that is because although there are still some things to work out with the game, the AI is a bit further on in development.  Whereas in GC2 I can crank it up to Tough (which is the highest difficulty before the AI starts to get bonuses) and I'll know that I can win, in FE at Challenging I can regularly get my butt handed to me.  So you must be doing something right there.

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 9:49:52 AM from WinCustomize Forums WinCustomize Forums

I wonder about the demographics of the responses (age, sex, etc.) and how each demographic rated the individual games (granted similar sampling sizes).

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 10:52:16 AM from Stardock Forums Stardock Forums

Quoting Lord Xia,
I've got some experience with statistics (I had to take it in college and did mediocre in it).  This isn't a good graph to really show a "trend".  The scale should go from 0 to 100.  Also, there are too few scores.  Also, reality check.  Getting a game review in the 90+ is exceptionally rare, and you simply can't consider that the standard to which you should compare all other released games.  First game is a 95, all other games should be 90+.  Prepare for disappointment.  Should be proud of anything in the green, 76+.     

Assuming all games were rated under the same criteria, the scale is fine, and a 100 point scale wouldn't be necessary to "normalize" the data. I'd say 8 data points is sufficient to assess a trend. (A statistically valid conclusion and a trend are two different things). The graph covers 5 years of activity. Just hoew many more would you need? In my business 6 data points (6 months) is enough to have confidence in a trend. (I don't mean confidence in the statistical sense)

But again, I'm not focusing on the individual scores. When you see a graph heading south, it's time to try to figure out why.

My guess is consumer expectations. A systematic risk that's mostly out of SD's control.

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 11:29:44 AM from Elemental Forums Elemental Forums

Two things:
The 'real' score of a game on metacritic can basically be measured between the aggregate scores of the reviewers and the aggregate scores of the user base. Most of Stardock's games are in line with one another while having reasonably good scores. Take a look at some other "Triple A" titles and tell me if you can say the same. Stardock doesn't have hundreds of millions to sink into "advertising heh-heh campaigns'.

Secondly, find some of the most well regarded RTSes [or any other game] that has ever been released and ask yourself how they would fare in reviewing circles if they were released today.

All this graph really suggests to me is people like Stardock's sci-fi games more than their fantasy games. Another interesting point of data you might consider adding to that graph is the trend of metacritic scores overall during the same time period.

 

 

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 11:38:25 AM from Elemental Forums Elemental Forums

Seems to me the only thing you can learn from that, is that the majority of the review scores used by MetaCritic, were pulled out of the reviewers hats.

I'm not entirely sure if that's a good thing for you guys, though. Because I'd have rated most of those releases lower. And hey, 5'ish for E:WoM is extremely generous. At release it wasn't just a terrible game, it wasn't in a playable state. It should have gotten a 1

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 12:22:59 PM from Elemental Forums Elemental Forums

I also think reviewers are being more critical now than in past years.  For a long stretch certain magazines were giving AAA titles super high scores that were entirely undeserving of those scores.  At the time I could only guess the scores were inflated because the magazine is under pressure to keep advertising dollars.

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 3:23:49 PM from Stardock Forums Stardock Forums

Quoting Simsum,
Seems to me the only thing you can learn from that, is that the majority of the review scores used by MetaCritic, were pulled out of the reviewers hats.

Well since he used the Metacritic data in his OP, it seems he believes the data is meaningful.

I think we all agree that WoM was an anomaly...didn't need a graph to see that.

So I assumed he made the post for a deeper reason other than an opportunity to quote a song from Sesame Street.

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 5:06:59 PM from WinCustomize Forums WinCustomize Forums

Personally I think it is because of PlayStations, WII's, xBox'es etc.  No one plays games on the PC much any more.  My son even got rid of all of his PC games years ago (when he was 15) and has never even thought of buying a PC game since he got his PlayStation.

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 5:41:18 PM from WinCustomize Forums WinCustomize Forums

It's the economy.

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 7:30:41 PM from WinCustomize Forums WinCustomize Forums

Expectation versus reputation can account for the 'trend' which cannot really be considered of statistical value as it's too small a sample to be truly viable....

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 9:06:59 PM from Elemental Forums Elemental Forums

I don't think it's fair to compare Twilight to FE- Twilight is a 2nd expansion.

Comparing GalCiv II vanilla to FE might be better.

Also, FE was amibitious, perhaps overambitious.

 

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 9:25:34 PM from Elemental Forums Elemental Forums

Quoting Jafo,
Expectation versus reputation can account for the 'trend' which cannot really be considered of statistical value as it's too small a sample to be truly viable....

 

A trend has nothing to do with statistics..

pattern of gradual change in a conditionoutput, or process, or an average or general tendency of a series of data points to move in a certain direction over time, represented by a line or curve on a graph.

You don't need a "sufficient sample size" to assess the general direction something is going.

For example. If you saw a graph showing 6 months (6 data points) of steadily declining (seasonally adjusted) sales, would you argue that there wasn't enough data points for it to be statistically valid and take no action, or would you try to figure out why the sales were declining?

If you waited for 30 months (30 data points) to assess the sales trend, it could be too late...


Heck, in every company I've worked for, 3 months of declining drivers was sufficient to raise a red flag.

Reason for Karma (Optional)
Successfully updated karma reason!
February 4, 2013 9:34:51 PM from WinCustomize Forums WinCustomize Forums

Quoting Borg999,
A trend has nothing to do with statistics..

Nice try.  How about 2 points ...first at 100 second at 50 ...which one is an aberration perhaps?  Perhaps both...perhaps neither.

Only a fool would suppose a third point would be zero ....but it's a "trend".

 

Take your 6 points/months...it may just be 'off-season' [commercially] and exactly the same trend if you look at previous [or later] years.

6 months of reducing sales/numbers/whatever may be actually NORMAL.

Statistics are required to be sampled correctly to be legitimate.  They CANNOT be sampled to suit one's argument.

Lies....damn lies...and statistics.

Reason for Karma (Optional)
Successfully updated karma reason!
February 5, 2013 8:54:41 AM from Stardock Forums Stardock Forums

Quoting Jafo,



Quoting Borg999,
reply 17
A trend has nothing to do with statistics..


Nice try.  How about 2 points ...first at 100 second at 50 ...which one is an aberration perhaps?  Perhaps both...perhaps neither.

Only a fool would suppose a third point would be zero ....but it's a "trend".

 

Take your 6 points/months...it may just be 'off-season' [commercially] and exactly the same trend if you look at previous [or later] years.

6 months of reducing sales/numbers/whatever may be actually NORMAL.

Statistics are required to be sampled correctly to be legitimate.  They CANNOT be sampled to suit one's argument.

Lies....damn lies...and statistics.

Sorry nice try yourself.

First, I specified "seasonally adjusted" because I new your were going to try to make that arguement. That is, seasonally adjusted numbers are normalized.

In regard to your 100/50 comment....In finanacial analysis, if you encounter 2 extreme numbers in a trend, the first thing you do is look at the accounting records. Did the staff accountant transpose numbers? Was there an adjusting entry from a prior period that was posted in the current perid, etc.

If that doesn't yeild results, then you look at the drivers. (the volume of something that driver sales - e.g number of units sold) -  Did a mistake occur in the recording of those drivers? If not, you go to the ops folks and ask? Was there a supply disruption? An external event out of the company's control? did a customer terminate a contract? is this a one time event? Do you expect it to continue, etc?

Typically, at the beginning of each year, a budget is built based on expectations for the next year that is based on economic conditions and company events known at the time. To standardize the process, dollar amounts are built on a rate/volume basis (e.g.units x sales per unit, or job hours worked x hourly rate, etc.). If the actual results deviate from the budget, closer examination is warranted. This analysis is typically done on a monthly basis.

Only a fool allows downward trends to continue for more than three months ("until there is enough data") without examining the underlying cause.

I seriously hope you're not a member of SD's Finance team....

 

 

Reason for Karma (Optional)
Successfully updated karma reason!
February 5, 2013 9:18:43 AM from WinCustomize Forums WinCustomize Forums

Quoting Borg999,
Only a fool allows downward trends to continue for more than three months ("until there is enough data") without examining the underlying cause.

Again...a 'downward trend' can be entirely normal...particularly if it's ONLY 6 months as your example....because the following 6 months....rest of the year may have a NORMAL uptrend.....

Think seasonal sport or such similar.

So....this 'fool' can ignore the 'downward trend' because he knows it is a normal function of his business.

As I said....don't expound on statisitcs as if it's God's gift to economics...it's simple smoke and mirrors and can be made to represent anything you want it to.

Lord knows who you are trying to educate re stats/maths/arithmetic...but I gave all that away back in '71 ...... since then it's all been numbers re building design/construction.....though I'm fairly confident numbers did the same crap in the 70's as they do now....

 

BTW....just to be clear...all this is hypothetical as I am not on SD's accounting staff...and either way would NOT be detailing the commercial activities of SD ...as it's not my place to do so...

Reason for Karma (Optional)
Successfully updated karma reason!
February 5, 2013 9:40:35 AM from Stardock Forums Stardock Forums

Quoting Jafo,



Quoting Borg999,
reply 19
Only a fool allows downward trends to continue for more than three months ("until there is enough data") without examining the underlying cause.


Again...a 'downward trend' can be entirely normal...particularly if it's ONLY 6 months as your example....because the following 6 months....rest of the year may have a NORMAL uptrend.....

Think seasonal sport or such similar.

So....this 'fool' can ignore the 'downward trend' because he knows it is a normal function of his business.

As I said....don't expound on statisitcs as if it's God's gift to economics...it's simple smoke and mirrors and can be made to represent anything you want it to.

Lord knows who you are trying to educate re stats/maths/arithmetic...but I gave all that away back in '71 ...... since then it's all been numbers re building design/construction.....though I'm fairly confident numbers did the same crap in the 70's as they do now....

 

BTW....just to be clear...all this is hypothetical as I am not on SD's accounting staff...and either way would NOT be detailing the commercial activities of SD ...as it's not my place to do so...

I don't think you are reading my full post... I already took normalization/seasonailty into account. If the decrease is know ahead of time (as built into the budget), then yes it can be ignored, but if it's outside the norm (expected results), then it should be examined.

I never said statistics was "God's gift". As a matter of fact, I was arguing, statistcs are not needed to assess a trend. Either you confused my post with someone else, not reading carefully, or are intentionally trying to be difficult.

Reason for Karma (Optional)
Successfully updated karma reason!
February 11, 2013 5:24:38 PM from JoeUser Forums JoeUser Forums

LONG LIVE GALCIV AND TBS!!!

Reason for Karma (Optional)
Successfully updated karma reason!
Stardock Forums v1.0.0.0    #108433  walnut3   Server Load Time: 00:00:00.0000656   Page Render Time:

Home | About | Privacy | Upload Guidelines | Terms of Service | Help
WinCustomize © 2014 Stardock Corporation. All Rights Reserved.