Git Product home page Git Product logo

sc2-pulse / sc2-pulse Goto Github PK

View Code? Open in Web Editor NEW
19.0 1.0 2.0 8.02 MB

The fastest and most reliable ranked ladder tracker for StarCraft2 (Spring Boot app)

Home Page: https://sc2pulse.nephest.com/sc2/

License: GNU Affero General Public License v3.0

Java 76.87% JavaScript 12.75% HTML 6.93% CSS 0.80% PLpgSQL 2.65%
starcraft2 spring-boot webapp api statistics battlenet discord postgresql reactor maven twitch

sc2-pulse's Introduction

About

SC2 Pulse is the fastest and most reliable ranked ladder tracker for StarCraft®2. It is a Spring Boot web application. The reference website (runs the latest release) is available 24/7. The public discord bot is also available.

Disclaimer

This application uses Battle.net® API. This is not an official Blizzard Entertainment® application.

Warning

Please do not use this project as a reference if you are new to programming.

This is my first Spring Boot project, and it's old, there is a lot of legacy code that is poorly though out.

I use this project to learn new stuff, so it's just a bunch of helloworlds of different technologies that evolve over time. Furthermore, I'm only interested in backend stack, so the frontend part of the project is a mess. I focus on new features and I don't have time to rewrite the old code. The project will remain as it is until I implement all the planned features.

Dependencies

  • Java 17+
  • PostgreSQL 15+ with btree_gist extension
  • Maven 3
  • BattleNet API access keys(you must use your own keys)
  • Twitch API keys(you must use your own keys)
  • Aligulac API key(you must use your own key)

Testing

Run the tests to ensure that you have a valid environment set up. You must also pass the tests before creating a PR.

A real PostgreSQL database with btree_gist extension is required for some integration tests. This should be only a testing db, as tests will drop/create the schema. Do not use your real DB.

You can use the src/test/resources/application-private.properties file (ignored by git, used by a test config) to create a simple test config:

spring.datasource.username={name}
spring.datasource.password={pasword}
spring.datasource.url=jdbc:postgresql://localhost:5432/{test_db_name}
spring.security.oauth2.client.registration.sc2-sys-us.client-id={client_id}
spring.security.oauth2.client.registration.sc2-sys-us.client-secret={client_secret}
spring.security.oauth2.client.registration.sc2-sys-eu.client-id={client_id}
spring.security.oauth2.client.registration.sc2-sys-eu.client-secret={client_secret}
spring.security.oauth2.client.registration.sc2-sys-kr.client-id={client_id}
spring.security.oauth2.client.registration.sc2-sys-kr.client-secret={client_secret}
discord.token={token}
spring.security.oauth2.client.registration.discord-lg.client-id={client_id}
spring.security.oauth2.client.registration.discord-lg.client-secret={client_secret}
twitch.client-id={client_id}
twitch.client-secret={client_secret}
com.nephest.battlenet.sc2.aligulac.api.key={api_key}

To run all the tests execute the following command in a terminal

mvn verify

Selenium tests

The Firefox is used in selenium tests because it is one of the major browsers that is available on all platforms and has an ESR version, which makes it easier to find a correct selenium driver for it.

You can change the browser by modifying the selenium.driver application property.

Running

The dev profile will help you to start the local server. Reload a browser tab to instantly see resource modifications. Build project to hotswap(if possible) the new classes.

Required application properties

You must set the following application properties:

server.port={port}
spring.datasource.username={name}
spring.datasource.password={pasword}
spring.datasource.url=jdbc:postgresql://localhost:5432/{db_name}
spring.security.oauth2.client.registration.sc2-sys-us.client-id={client_id}
spring.security.oauth2.client.registration.sc2-sys-us.client-secret={client_secret}
spring.security.oauth2.client.registration.sc2-sys-eu.client-id={client_id}
spring.security.oauth2.client.registration.sc2-sys-eu.client-secret={client_secret}
spring.security.oauth2.client.registration.sc2-sys-kr.client-id={client_id}
spring.security.oauth2.client.registration.sc2-sys-kr.client-secret={client_secret}
spring.security.oauth2.client.registration.sc2-lg-eu.client-id = {client_id}
spring.security.oauth2.client.registration.sc2-lg-eu.client-secret = {client_secret}
spring.security.oauth2.client.registration.sc2-lg-us.client-id = {client_id}
spring.security.oauth2.client.registration.sc2-lg-us.client-secret = {client_secret}
spring.security.oauth2.client.registration.sc2-lg-kr.client-id = {client_id}
spring.security.oauth2.client.registration.sc2-lg-kr.client-secret = {client_secret}
discord.token={token}
discord.bot.invite.url={discord_bot_invite_url}
spring.security.oauth2.client.registration.discord-lg.client-id={client_id}
spring.security.oauth2.client.registration.discord-lg.client-secret={client_secret}
twitch.client-id={client_id}
twitch.client-secret={client_secret}
com.nephest.battlenet.sc2.aligulac.api.key={api_key}

Optional application properties

The server will run without errors if these properties are missing, but the corresponding features will be disabled.

com.nephest.battlenet.sc2.replaystats.api.key

Misc

You can use the src/main/resources/application-private.properties file (ignored by git, used by config) for private/local application properties

Use DB init script to create a fresh DB.

To run the local server execute the following command in a terminal

mvn spring-boot:run

Scheduled tasks are disabled in the dev mode. You can remove the @Profile annotation from the Cron class if you want to run the tasks(like ladder scans) in the dev mode.

Alternative update

Legacy and profile ladders

The Blizzard API can sometimes break and return stale data. The app checks the API state before every update and will switch the endpoint route if any problems are found. This happens automatically and individually for each region, so you can have a situation when KR region uses the alternative route, while other regions use the usual route.

Alternative update limitations:

  • no league tiers
  • no BattleTags
  • partial racial info
  • slower update

The missing info can be fetched from the main endpoint when it's back up(it happens automatically).

Original idea by Keiras

Forced API host

You can manually remap some endpoints to use another API host. This is useful when one region is broken but others are not. See addForceAPIRegion method of the AdminController. Set application property com.nephest.battlenet.sc2.api.force.region.auto to true to enable auto remap algorithm.

Web API

The Blizzard web API can be used as a last resort when everything else breaks. It is disabled by default. Some endpoints can be manually redirected to web API via AdminController. You can enable auto web API by setting the com.nephest.battlenet.sc2.ladder.alternative.web.auto application property to true.

Blizzard ToS compatibility

Even though the API is not forbidden via robots.txt, the Blizzard Developer API Terms Of Use clause directly forbids it

You May Not Data Mine Blizzard Products Or Services. Except as permitted through authorized use of the 
Blizzard Developer APIs, You will not perform any data-mining, scraping, crawling, or use any processes that sends 
automated queries to Blizzard or any Blizzard game, service, or website, or use any other similar methods or tools 
to gather or extract data other information from Blizzard or any Blizzard game or service.

To ensure that the potential violation is a minor one, the following rules are applied:

  • a very low request rate is used
  • the data is considered as if it came from the regular dev API and the relevant ToS and Privacy Policy are applied.

Use it at your own risk.

Task configuration

Cron class contains all scheduled tasks.

Application properties

  • com.nephest.battlenet.sc2.url.public public URL of your service
  • com.nephest.battlenet.sc2.mmr.history.main.length 1v1 mmr history length in days, 180 by default.
  • com.nephest.battlenet.sc2.mmr.history.secondary.length team mmr history length in days, 180 by default.
  • com.nephest.battlenet.sc2.ladder.regions set of regions that will be updated, currently active regions by default.
  • com.nephest.battlenet.sc2.db-dump-file path to the database dump.
  • com.nephest.battlenet.sc2.api.request.limit.separate Activates legacy mode where each region uses a separate request limiter. False by default.
  • com.nephest.battlenet.sc2.privacy.character.profile.update update characters directly from their bnet profiles. True by default.
  • contacts.email
  • contacts.discord.server
  • contacts.discord.server.name
  • contacts.twitter
  • contacts.github
  • security.remember-me.token.key key for signing remember-me tokens
  • security.remember-me.token.max-age Max age(duration) of the cookie, P3650D(~10 years) by default

Contributing

Want to make a bug report/feature request? Any contributions are welcome, see CONTRIBUTING for more information.

Licenses

Blizzard ToS

SC2 Pulse is fully compliant with the Blizzard ToS.

  • Requests per hour cap is guaranteed if your clock desync is within 5 seconds.
  • Requests per second cap is guaranteed.
  • BattleTags, player names, and matches are removed after 30 days from the moment they were deleted from the API

Trademarks

Battle.net, Blizzard Entertainment and StarCraft are trademarks or registered trademarks of Blizzard Entertainment, Inc. in the U.S. and/or other countries.

sc2-pulse's People

Contributors

frugs avatar nephest avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

Forkers

leisvan frugs

sc2-pulse's Issues

Create custom error pages(thymeleaf)

  • Errors to handle: 404, 500
  • Template engine: thymeleaf
  • Template: base
  • Fragment: generic error
  • The page must contain
    • {errorCode} - {errorText} as title and header
    • Descriptive error text
  • Do not create an additional controller for this, use thymeleaf's predefined error pages

Improve loading speed of the profile page

Originally the profile was intended for 4 months of MMR history. Currently we have 3+ years of MMR history and some profiles can take a long time to load.

  • Switch to lazy loading of each tab
  • Improve MMR history performance
    • Don't serialize the history into arrays
    • Don't join on duplicate data, starting from divisions
    • Look for bottlenecks in the JS code, fix them if possible

Personal settings/BattleNet integration

Give users an ability to log in using the BattleNet oauth flow. See if there is some interesting personal information could be fetched from the API.

Give users an ability to bookmark favorite teams. Display a list of all bookmarked teams.

Show count of bookmarks on the ladder?

#8 must be done first

Better player profiles

Profile should contain:

  • battlenet profile picture (#13 (comment))
  • battletag
  • battlenet profile link
  • player history
  • stats

Profile stats section should contain summary sections for all team types, formats, races (where relevant data is present). These sections must include:

  • games played
  • highest MMR
  • highest league

This data should potentially be precalculated in the player_character entity. It will enable us to use this data in other locations(like character search).

Optimize queries

Postgres migration is complete, it is time to fully leverage its advantages. Main ladder search query should be optimized. It can also lead to pagination rewriting. Target execution time is under 100ms.

Rewrite frontend

The current frontend code is bad to say the least. There are many reasons for this, so I'll describe the most notable one.

The main goal of this project for me is to learn java web development(Spring in particular). When I first started developing the project, it was nothing more that one page with sorted ladder. I have very little experience with JS, but it was enough since the fronend part was close to nonexistent. As the project features expanded, so did the frontend code. But I was still using the vanilla JS and learned along the way, so the code base has very little coherence as the result.

If you are currently learning the JS/frameworks, you are free to use this project as a practice field. You can rewrite the frontend completely from the scratch and use whatever technologies you like.

Requirements:

  • simplicity
  • responsive&mobile friendly design
  • use /api/* endpoints to get data
  • tests

Equal battletags are counted as distinct entities in the league stats

Even though accounts with equal battletags represent different entities from the bettlenet standpoint, they still represent the same player that just happens to play in a few different regions. We could just merge such accounts in one singular entity, but account in different regions could potentially have some unique settings. LeagueStatsDAO should count accounts with distinct battletag instead of distinct id.

Fix loading splash shaking

Loading splash (#status-generating-fullscreen) spergs around if a scrollbar is appearing during loading/generation.
Some info:

  • CSS only

CN BattleNet links are invalid

Problem

Profile(frontend popup) BattleNet links from the CN region are leading to 404 result on the upstream website. Looks like CN profiles are available only for CN users somewhere behind their firewall. Remove BattleNet profile links for the CN region.

Solution

Place to start: showCharacterInfo js function (found in CharacterUtil.js)
How to fix: toggle visibility of #battlenet-profile-link depending on the region(hide if CN, show otherwise). Visibility is controlled by the Bootstrap class d-none;
Notes: Region enum can be found in the enum.js

Twitch links don't update automatically

The daily reverse lookup by id doesn't work, modified URLs don't update. There are no errors in the log, most likely it just fails to run. Might need to check the debug log and add more debug messages if needed.

Clarify Blizzard's 30 days information removal policy

Blizzard requires downstream users to remove all data that is no longer available via their API in 30 days. Blizzard also stated that leaderboards are considered public data and is not a subject to this rules. Do we need to follow the 30 days policy if we fetch only the ladder data(no personal profile data)?

Links:

Solutuion: in the original thread a bunch of people have already asked similar questions. That resulted in a bluepost update regarding leaderboards data usage(section 15). It seems that as long as we do not fetch anything except the ladder data we are free to keep it forever.

Infinite scroll

Add an ability to call a js function when the viewport is about to be scrolled down to the end of the target element/page.

  • The target element is an element that contains paginated content that can be extended by loading the next page.
  • A "Load more/next" control can be used as the target element if it already exists on the page. Such elements are expected to handle pagination logic like navigation, completion, and reset, so you can just observe these elements via the Intersection Observer API.
  • Use the Intersection Observer API

Here are some content/functions that can be enhanced by this feature. Notice that these functions expect the evt parameter to be present, so add a null check or pass an empty event object.

static loadNextMatches(evt)

static loadNextMatches(evt)

static loadNextMatches(evt)

You can register the observers here

Be sure to modify the Selenium test to test endless scrolling instead of button clicks.

private static void testCharacterMatches(WebDriver driver, WebDriverWait wait)

BlizzardSC2API full fetch integration test

Modify BlizzardSC2APIIT testFetch test to do a full fetch, from season to team member. See the StatsService updateLeagues method for logic reference. You need to do only the fetching part, no need to actually write anything to the DB.

Fetching order:
1 season
2 league
3 league tier
4 ladder

You must verify (with Validator) that received entities are valid. See season validation in testFetch method.

One fetch chain is enough, you need to fetch only a single ladder, no need to scan the whole season.

Audit log for revealers

Data

  • datetime
  • user(empty for system)
  • action(delete, update, insert)
  • affected entities
    • pro player
    • character
  • JSON diff
  • full JSON entity

Website

  • API endpoints with filters
    • user
    • action
    • affected entities

Discord bot

  • Post new records in community server, private channel
  • Mention revealers(authors) when possible

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.