A way to make it only notify if a character cannot be added to the database would be useful for me (but, it's just a request, no worries :D)

Hey guys,

do you have some more recent statistics on how long does it take to run for the entire base?
How many characters are you gathering per seconds/minutes?
What are the hardware you are using to run it?
Bandwidth consumption?
CPU, MEM usage?

I'm curious on the performance of the current implementation, also, would be very good to provide this information in the README, so other people can have a notion of what they can expect for the current version, especially a more recent "high id", so it won't run forever, and also give some information so others can help you improve it :)

Anyway, good work doing it, it's very interesting.

I decided to run this collector myself because it looks like ffxivcensus.com data is pretty old, have it running on an EC2 instance in japan and it's chugging along fine but I'm really just curious, did SE really not block your IP??

This doesn't seem to care about any sort of rate limiting

java -jar XIVStats-Gatherer-Java.jar 1 13500000

Connection failed! Please see output console
java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/dbplayers
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:215)
at com.ffxivcensus.gatherer.GathererController.openConnection(GathererController.java:234)
at com.ffxivcensus.gatherer.GathererController.main(GathererController.java:81)
java.lang.NullPointerException
at com.ffxivcensus.gatherer.GathererController.main(GathererController.java:83)

I have JDBC connector installed. Running Ubuntu on a VPS.

Taking a look at the test results and an example character it looks like the data centre is now being included in parentheses after the realm name. This is going to cause issues with the realm count as @Pricetx has noted to me offline.

The layout of the lodestone has changed, unit tests are now failing indicating gatherer can no longer correctly parse page!

Resolution of this issue is a high priority.

Thanks go to @fahy for notifying me of the possible impact of this!

Out of curiosity, when the Oceania Servers and DC get released, would this automatically be accounted for or would be a feature item?

Completion of A4S guarantees the Gobwalker mount.

Completion of A8S guarantees the Faustlet minion.

Please check spelling

I was beginning to debug the active player count on ffxivcensus.com by running the latest version of the java gatherer in a test environment.

When running the gatherer on my character (8308898) there appears to be a number of empty fields. I have attached a CSV showing an excerpt (please ignore the formatting with the mounts, limitation of csv).

The following fields all appear to be blank:

• p30days
• p60days
• p90days
• p180days
• p270days
• p360days
• p450days
• p630days
• p960days
• prearr
• prehw
• presb
• arrartbook
• hwartbookone
• hwartbooktwo
• hasencyclopedia
• beforemeteor
• beforethefall
• ps4collectors
• soundtrack
  -saweternalbond
• sightseeing
• comm50
• moogleplush
• topazcarbuncleplush
• emeraldcarbuncleplush
• hildibrand
• dideternalbond
• arrcollector
• kobold
• sahagin
• amaljaa
• sylph
• moogle
• vanuvanu
• vath
• arr_25_complete
• hw_complete
• hw_31_complete
• hw_33_complete
• sb_complete
• legacy_player

Some of these fields may intentionally be blank as my character doesn't meet all of the requirements, however, a number of them should be populated.

The arguments used when calling the gatherer:
/usr/bin/java -Xmx1024M -jar XIVStats-Gatherer-Java-v2.0.0.jar -s 8308898 -f 8308898

The version of the gatherer is the most recent build on ci.reidweb.com. This is build 78 and is built from commit 4f9f2c8, which at the time of writing is the latest commit to master.

I have tried re-creating the MySQL database in case that was somehow an issue, but this did not affect the result.

It does not look like this is the issue currently facing the live environment as it has data for the soundtracks, collectors editions etc, so I will continue investigating that.

8308898.xlsx

When parsing a new character, the Mounts & Minions page have no content, so the there's a silent error caught in the logs. It would be better to detect and mitigate for this rather than throw an exception.

[2020-02-26 11:24:59,503] [pool-1-thread-29] DEBUG com.ffxivcensus.gatherer.task.GathererTask - Starting evaluation of player ID: 19066937
[2020-02-26 11:24:59,753] [pool-1-thread-3] ERROR com.ffxivcensus.gatherer.task.GathererTask - Index: 0, Size: 0
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

Found during testing of #57

It looks like the lodestone has again separated out some more content into different static pages, so we'll need to update the class & job level parsing to cope with this.

With the release of 4.5, we'll have BLU available as a new job. We'll need to add stats for BLU and update the lodestone gathering when it's updated.

Unit tests need to be implemented for the 'Console' class before final release of v1.0.

The latest lodestaone change moved Mounts & Minions into separate pages, which means that we have a lot of missing stats and our active counts are screwed.

Gatherer needs to be updated to handle the new layout.

Currently, the application outputs the results of what it's doing for each character to STDOUT. It would be nice if in addition to this I could have it output to a file so that I could then make some sort of progress bar in a webpage.

I can probably split STDOUT into multiple locations in the shell, but having a log file option just seems like it would be generally useful.

Wanted to set up a new site with the support of new jobs (recently introduced). All went well, but finding the highest ID was a bit tedious. I had to hand edit the number and play around for an hour. And still, I am not sure if I have the highest valid ID at all.

Would it be possible to add a way to the Java collector to find the highest ID?
Thank you.

Example, why it would be useful:

My script is at: 1,803,264
The fresh stat page (mine) says: 242,560 players.
According to my hand guess, the highest ID is: 19,096,969
Meanwhile, the "official" site lists 10 million players, So at this rate, my script is really off.

The structure for tblplayers listed on the main project page is out of date with the most recent SQLdump (2018-01).

It is now as follows:

columns = [
           'id',
           'name',
           'realm',
           'race',
           'gender',
           'grand_company',
           'free_company',
           'level_gladiator',
           'level_pugilist',
           'level_marauder',
           'level_lancer',
           'level_archer',
           'level_rogue',
           'level_conjurer',
           'level_thaumaturge',
           'level_arcanist',
           'level_darkknight',
           'level_machinist',
           'level_astrologian',
           'level_scholar',
           'level_redmage',
           'level_samurai',
           'level_carpenter',
           'level_blacksmith',
           'level_armorer',
           'level_goldsmith',
           'level_leatherworker',
           'level_weaver',
           'level_alchemist',
           'level_culinarian',
           'level_miner',
           'level_botanist',
           'level_fisher',
           'p30days',
           'p60days',
           'p90days',
           'p180days',
           'p270days',
           'p360days',
           'p450days',
           'p630days',
           'p960days',
           'prearr',
           'prehw',
           'presb',
           'arrartbook',
           'hwartbookone',
           'hwartbooktwo',
           'hasencyclopedia',
           'beforemeteor',
           'beforethefall',
           'ps4collectors',
           'soundtrack',
           'saweternalbond',
           'sightseeing',
           'comm50',
           'moogleplush',
           'topazcarbuncleplush',
           'emeraldcarbuncleplush',
           'hildibrand',
           'dideternalbond',
           'arrcollector',
           'kobold',
           'sahagin',
           'amaljaa',
           'sylph',
           'moogle',
           'vanuvanu',
           'vath',
           'arr_25_complete',
           'hw_complete',
           'hw_31_complete',
           'hw_33_complete',
           'legacy_player',
           'mounts',
           'minions'
          ]

All of these dumps seem to be missing the two most interesting features: date_active and is_active.

I'm doing a pet project using topic modeling of Lodestone forum post titles against various data points, namely active subscribers. I could key these counts manually from the ffxivcensus site, but where's the fun in that? 😄👍🏼

So looking at LuckyBancho's stats that they recently released here: https://www.reddit.com/r/ffxiv/comments/apfx16/luckybancho_unofficial_census_10_february_2019/

It seems that they are tracking for Elemental Levels (it's under the Class/Job tab on a character's page)
As of Patch 4.55 (Released today 12th Feb 2019), there are 4 "Eureka" instances, with there being Level caps for each zone. Now it is possible to level past 20 in Anemos but I figured that the figures could be displayed in the next (March report) once players have got to max level in Hydatos.

• Eureka Anemos - 1 - 20
• Eureka Pagos - 21 - 35
• Eureka Pyros - 36 - 50
• Eureka Hydatos - 51 - 60

My idea would be that the result would be:

• "Total Players who have played Eureka": (eg. Everyone that has a Elemental Level)
• "Players who are levelling in Eureka: Hydatos" (eg. Anyone that is between 50 and 59)

Add Red Mage and Samurai to parse list.

Verify that all other jobs are being correctly evaluated - I cannot remember off the top of my head if they are index based or HTML text based on how it determines which is which.

We have just completed a new census run for December 2021. The stats show a remarkably low number of characters. On further inspection, the number of DELETED characters seems extremely high:

SELECT COUNT(*) FROM tblplayers;
34,329,901

SELECT COUNT(*) FROM tblplayers where character_status = "DELETED";
28,142,672

The result is that the census reckons there are only 6,187,229 characters in existance.

We should probably investigate and try to find some examples of characters the database reckons are deleted but actually aren't to enable us to troubleshoot further.

For the past month or so the run has been terminating early, at around ID 1.5 million. It was previously thought that it was an environmental issue. However, further investigation has uncovered an issue with a system designed to detect the end of the player list.

There is a system in place which detects when 200 consecutive IDs do not have characters associated with them. When this condition is detected, the run is terminated. However, this is experienced both at around 1.5 million and at around 1.6 million in the ID range, example shown below:

[2019-02-11 18:50:42,571] [pool-1-thread-16] INFO com.ffxivcensus.gatherer.player.PlayerBuilder - Character 1606804 does not exist. (404)
[2019-02-11 18:50:42,577] [pool-2-thread-1] DEBUG com.ffxivcensus.gatherer.task.LevemeteTask - Loading new jobs into the gatherer
[2019-02-11 18:50:42,610] [pool-2-thread-1] DEBUG com.ffxivcensus.gatherer.task.GatheringLimiterTask - GATHERING CAPPING: Checking whether the gathering should stop...
[2019-02-11 18:50:42,613] [pool-2-thread-1] INFO com.ffxivcensus.gatherer.task.GatheringLimiterTask - GATHERING CAPPING: FINISHING - No valid characters found for at least 200 ID's after Character #1606512

for safety, I am going to raise this number to 5,000, as this still only represents a few minutes of scanning beyond the last character but it much less likely to accidentally trigger like it has done here.

As the main story has you choose a Grand Company, I found it odd that there were active characters (with the Endwalker MSQ minion) with no Grand Company. Digging a bit deeper, I found that characters in the database with that minion and their grand_company field set to 'none' had one thing in common; having a PvP team listed on their Lodestone page. Admittedly, I only checked a couple IDs, but after a dozen or so I assumed the problem extends to all 17,145.

I think the issue stems from characters on PvP teams having an additional character-block field on their Lodestone page, causing both the Free Company and Grand Company functions in the PlayerBuilder class to fall through when the number of elements is 6, rather than 3-5. I also assume the case of having 5 elements could refer to a character with a PvP team but no Free Company and cause an issue somewhere. I don't think a character can have a PvP team with no Grand Company, but I'm not 100% certain on that, as I don't know exactly when PvP is unlocked.

If you'd like the full list of IDs, the query I ran was:
SELECT id FROM tblplayers WHERE grand_company = 'none' AND minions like '%Wind-up Herois%';

For simplicity, here's the first 5 results so you can check their pages:
5313
6855
6913
8283
9307

With the switch to Spring Boot, the Command-Line error handling is a lot less obvious and can get hidden in Spring's startup sequence.

Need to investigate how to pull the CLI parsing forward in the startup sequence and make sure the actual errors are more obvious.

README.md needs to be updated in line with changes to console interface implemented recently.