Another Weekly Overview fix. The subdomain versions (DS, PSP, Kindle) weren't generating properly, but now should be.
Fix to the Weekly Overview. Last time I meddled with it I changed how it handled the // URLs, but it tried to use a function before the file containing said function was include()d. OOPS.
So today I made the Hardware Shipment Hypothetical Generator and stuck it in /labs. It's a modification on the Hardware Shipment Line Graph Generator and so already is more polished than most of the things sitting in /labs, but as it isn't about presenting sales facts and is instead about making hypothetical situations more easily, it's probably going to stay off the main page.
However, since it IS a modified version of old shipmentcompare, when I saw a few things that were wrong or could be improved a little, I then applied them to shipmentcompare proper. Most notably, when doing Date mode in the second HTML view option, the days since launch columns were waaaay off base due to a flub I made while making changes a week or two back.
So I noticed Weekly Overview was still using the old hardware tables, and went in to change it. While I was in there I did a little bit of cleanup, as well as added something new. Now along with the weekly sales of the individual systems, it will show the sales of hardware families and manufacturers. It only shows these if there's more than one system adding in to such a total in a week. So for the present the only hardware family shown is DS since Media Create still lists both DSi and DSiLL, and Microsoft isn't listed among the manufacturers since it's solely made up of the already-listed X360 number.
In a story of true perseverence, besttoworst.php has been upgraded from the /labs directory after sitting in there for most of the site's life. It used to run inexplicably slow--and while I still didn't exactly pinpoint the cause, I made enough changes that it doesn't seem like it's an issue anymore. At first I just wanted to make it use the new hardware table, but while I was at it I replaced the links to systems at the bottom with a series of dropdown boxes at the top cribbed from the Japanese Hardware Sales Line Graph Generator, and made it able to function with hardware families and manufacturers. Before, for example, it was hard to get a good grasp of DS's best and worst weeks through its life, because it had so many hardware models which heavily overlapped. Now, it's easy. In fact, hardware family is the default, so if you want just original model DS you'd have to tell it to use the System group type. Also, rather than Media Create and Famitsu being listed together as default, you pick which tracker you want the list from.
Some notable changes to an oldie, the Software Groups page. First, notice that it's now softwaregroups.php instead of softwaregroupid.php. This way makes more sense; the old way was just how I ended up naming it early in development and it suck. Second, it caches. Not that generating a pretty straightforward list like that is so taxing, but hey. Third, new columns have been added to give more information about the groups without having to enter their individual pages: the number of games contained, and their last known combined LTD sales. Fourth, rather than the Software Super Groups being listed underneath the regular Software Groups, they are now side-by-side--unless you're on DS. Fifth, for a while it's let you choose which of the two columns you want the information sorted by. The two new columns are options as well. However, also the reverse options are in--so if you've clicked to see software in alphabetical order, clicking on the Name link again will give you the reverse alphaetical order.
Japanese Software Sales Group Bar Graph Generator is back. Barely touched a thing. And with that, all the things I blocked off with this picture should be back to normal. Hopefully that's done some good as far as being too taxing with MySQL queries, but there are still a few other things I will tinker with.
Yesterday I forwarded old static Versus setups to the new generic Versus. I've similarly done the same for the old software.php that's been sitting around the server past its point of usefulness. It just lists the games a platform has. Sooo anyone trying to go there is just redirected to the Game Search. Since the ?platform=whatever is the same between the two, they'll even get the results they were looking for--more or less. The default order isn't the same, and the &week= attribute won't translate perfectly, but for as little as this is ever accessed anymore: close enough.
Hardware Shipments "In terms of" Generator also brought up to date. Luckily this one was much simpler--looks like my past self was doing things a bit smarter this time. Still, added in caching, shortened the image URLs a bit, other small tweaks. The generator frontend page has also had its dropdown boxes updated.
Separately, slight fix for linecompare.php. Nothing apparently wrong, but I noticed a few places where it was still referencing the old hardware tables rather than the new one.
The Japanese Hardware Sales "In terms of" Generator is back. Like the Versus earlier today, internally it was almost all replaced, though the end result should look the same. But the results are there where it mattered. Input for input, this must've been the most queriful script I had. Not kidding when I say that comparing the before and after for a fairly simple sample case, it went from 1037 to 4. Caching is in effect here, too.
Other than those changes for purposes of efficiency, there are a few minor fixes in there. Before it was a real stickler about demanding a start date for every system; now if given no date it defaults to the earliest data available. The date alias system first seen in linecompare is also an option.
The generator page itself has been updated, too. It was pointed out that Vita was missing from the dropdown boxes. That's no longer the case, and the updated split table format previously seen on a few other generator pages is in use, too.
I mentioned there was no Versus generator. Well, there is now. It's not much to look at, but it functions. One difference between this and the other generators is I left the system input as a textbox rather than a dropdown box. Disadvantage: If you're less nerdy into this than me, you may forget something is available or what the proper abbreviation is. Advantage: No needing to go through an increasingly-long set of options to pick the one you're looking for. I've been thinking of changing the others to this way; this is a bit of a test case to see which way is more annoying.
There's no simple HTML generator for them (yet), but the VS pages are among the older things on this site. After making a few that had no options (DS vs PSP, PS3 vs Wii, etc.) I made vsarbitrarytwosystems, so any two arbitrary systems could be used. Today I made that one a lot less query-greedy, and made many other internal changes as well, including--no surprise--caching. Admiral, this is an almost totally new Enterprise.
I also made it so all the old premade ones like PS3 vs Wii and 3DS vs PSV automatically forward a person to vsarbitrarytwosysems with the appropriate arguments given.
A bonus of caching more things is that by looking at the cache directory I can see what's being created. In this case I saw a bunch of things that definitely seemed like the visiting of machines. I think it's in general a good thing when a search engine crawls the site and helps bring it to someone who's looking for information, but there's no need for it to crawl, say, a list of all PS2 software sales through one date, and a date a few months after, and a date a few months after, and on and on. Anyway, I just made a few additions to the robots.txt that these things are supposed to abide by; hopefully that cuts down on their gathering useless info, and the related load on the server.
Softwaregroupline.php is a companion to linecomparegamejp.php. It's used at the bottom of group pages and takes in the group number and a bit of other information, then interprets that data and sends it on to linecomparegamejp.php. Especially since the advent of super groups, some groups are just so large that it just wigs out and gives no result. Others produce a result, but you just end up with a horrible line jumble that must be relatively intensive to create. Anyway, I've now limited it so it will only try to use the top 50 titles in a group. Enough for most groups, and more than enough to give a decent visual on the major titles for the rest.
Japanese software lines are back. Again many date-related tasks were changed to not use SQL, and again HTML results are being cached.
Shipment line graphs are back. This one was again full of me using MySQL for things I could instead do with PHP. Also, like linecompare before it, the HTML result views are being cached.
There are some things I was doing with MySQL that can be done perfectly well on PHP... at a slightly higher version than this server is currently using. There are workarounds to do the same thing (or close enough for my purposes), but it was starting to look cluttered with some of these little things repeated in several places. Anyway, I went ahead and made them into functions included in the Garaph Class every one of these files has access to. No difference to you, just a bit more organized for me.
Shipmentbar is back, with just a few minor changes. Mostly harmless.
Japanese hardware sales bars and pies are back. These siblings were always pretty simplistic from a database standpoint; upon confirming there wasn't much I could improve on in that regard, I just made some general improvements like switching them to using the new hardware table--the sooner they all use it, the sooner I can stop needing to keep the older one updated.
Doing these reminded me that I never actually turned off the software Top 30 bar chart, but that's OK: it was neither query-heavy nor in need of a change on something fundamental like the guys in the previous paragraph. Still, made some tiny invisible-to-you modernizing changes to it, since it hadn't been changed in about 2.5 years.
Found and fixed a little error in linecompare.php that I think and hope was the mysterious disappearing one I mentioned yesterday. It had to do with failing to read data from weeks where there isn't data yet or a system has stopped being listed.
Japanese Hardware Sales Line Graphs are back. The optimizations I made didn't really make it quicker, but it does use a loooot fewer database queries. The newer hardware table that contains cumulative sales listed weekly, as well as group and manufacturer numbers listed weekly, means it could pretty much stop needing a separate query for every single number. The graph images have long been cached, but now so are the HTML views.
Additionally, part of the HTML view 2 used a MySQL query for a simple bit of date manipulation that was simpler to do that way than in PHP. However, with a change of several years and versions this has been replaced with a PHP equivalent for even more savings.
For a while I was getting a MySQL error that I couldn't figure out, but it just went away. So it may be that there's an issue with some specific type of input I'm overlooking right now.
Game Search is back. Two pretty decent behind-the-scenes improvements.
1. When getting results from a database query, it's quicker to request a full row of data at once than an individual cell at a time. This may not make much difference when grabbing a few specific pieces of information, but when the search may return hundreds of results it can really add up.
2. Search results are cached. I didn't do this before because I didn't think it would make a very big difference. But now that I'm going through specifically to optimize things, no reason to ignore a pretty simple thing that might help some. This way if you return to look at the same type of results, or link someone else to results you've created, or multiple people do something simple and common like just list every PS2 game with no other options changed, it will have the results ready to go. Thanks to data coming in that could change results so often, though, these cache files will probably be wiped whenever the weekly Famitsu software data is added in anyway. You should be able to look at the bottom-left of the page and see if what you're getting comes from the cache or has just been put there.
The software group pages are back. No major internal changes needed this time. I did remove the links to seeing the state of groups in previous/next weeks. It seems something that's so little needed, all those links do is invite crazy robots to look at years' worth of weekly group data with no real gain. However, the URL format will still work. Depending on the URL form you're using just add something like //week/2004-11-29 or &week=2004-11-29. It will just be a "secret feature"; much as OrderBy already has been for the same page.
This is the last of the major mostly-static pages; next I get to start messing with the search and generators.
Today the individual software pages return. I took care of the header/exit thing from yesterday, but the bigger fix was something completely different.
The stuff with hardware attach rate that was only added a few months ago was a bit of a hassle. To account for multiple system models which may not all be listed in the same weeks (like the original model DS stopped showing up in weekly listings) took a bit of a crazy query per week--at least I couldn't think of a query that would give me all the results together. However, I went ahead and took the better long-term route today--the "hardware family" weekly and LTD numbers are now stored just like the individual systems are, so getting the data for all DS systems is no more trouble than getting the sales for one of them. This means each individual row doesn't require its own query, and games with a looot of weeks like the linked-to New Super Mario Bros. don't require hundreds.
I also increased the default time the individual software pages and weekly overview will stay cached. This should be a pretty invisible difference; with the way I have things set up now I pretty much clear out anything that should be changed each week, and if there's no new data coming in the page should be static. Some random week from 2005 or the listing for Tekken Tag Tournament probably shouldn't change unless I change the format of the pages, so whether they're set to cache for 30, 90, or 900 days should make little difference.
I figured the image showing up everywhere would explain the situation, but for those of me looking back at the archives in the future, here is said image.
The Weekly Overview is the first thing to return. The more static and able to be cached something is, the less taxing it should be. Even still, I was still getting some phantom results I wanted to work out before letting it back into the wild. If you tried to access a date that was too early or too late, it forwards you to the current week. So how was the cache still filling up thanks to robots trying to get Famitsu sales data from 1967? Here is your PHP PROTIP of the day! It turns out that even after the user was forwarded, since I didn't explicitly tell the script to end at that point it continued on and made a mostly-blank page. The next person/robot to try this bogus date would get the cached version of the mostly-blank page and follow it to the next bogus date, and so on. Now it should not. I've probably got similar trouble for other pages, but I have no time to mess with them today.