olismohn / aoppa Goto Github PK
View Code? Open in Web Editor NEWThis project forked from bitnykk/aoppa
Auno's AO DB parser fork
License: GNU General Public License v3.0
This project forked from bitnykk/aoppa
Auno's AO DB parser fork
License: GNU General Public License v3.0
AOPPA -- A program for extracting the AnarchyOnline resource database $Id: README 1113 2020-08-19 22:46:01Z os $ Copyright (C) 2002-2009 Oskari Saarenmaa <[email protected]>. All rights reserved. Updated by Bitnykk for 2020 compatibility Licensed under the GNU General Public License version 3. ============================================================================ Background, 'aoppa' is the current database parser I use to parse Funcom's CTree resource database and store it in a PostgreSQL database for use in my website (auno.org). This is the second, or perhaps third, generation of the parser, the first one was written entirely in PHP and had tons of problems. Including the fact that it took several hours to parse the database, in comparison it takes only about 5 seconds for aoppa to parse the current shadowlands database. As you can probably tell, this is mostly Unix stuff -- I don't do any development on Windows. The parser can only read a special data format that is produced by a Windows program that actually dumps the Anarchy Online CTree database. The program is called 'adbd' and the source code for it is available at http://auno.org/dev/adbd/ . I produced the program on a Linux system with the MinGW32 cross-compiler and run it under Wine. It would probably be trivial for someone familiar with Windows to rewrite such a program, the only thing it has to do is fetch records from the CTree database and then store them in a file in the format AOID: <id> LEN: <length>\n==========\n<data>\n==========\n Plugins, Aoppa supports plugins, in fact it can't do anything at all without at least one plugin loaded. Currently there exist two kinds of plugins, 'input' and 'output' ones. Input plugins are used to feed data to the parser, and output plugins are used to store the data structures populated by the parser. Aoppa ships with four default plugins, input-mmap and input-gzip for reading data from files generated by ADBD (see above), and output-xml and output-ign for writing XML and IGN-bot format output files. There's also the aunoorg-output plugin that I will discuss a bit later. The plugin system is quite simple to use if you wish to create new plugins, you should just add a public structure of the type aoppa_plugin_t called 'aoppa_plugin' in your code and then compile it as a shared object. The aoppa_plugin structure contains the following elements (in order): api version - for specifying the aoppa API version plugin type - can be AOPPA_INPUT, AOPPA_OUTPUT or both init function - function to be called when plugin is loaded finish function - function to be called when plugin is unloaded store function - function to store item_t structures flush function - function to be called when done with input file parse function - function to be called with a file to parse description - a const char * with a short description of the plugin All of these, except the version, are optional and can be NULL if the specified functionality is not required or wanted in the plugin. See aoppa.h or the supplied plugins for more information. The obvious plugin that people would probably want, and the one I would like to see developed is one for reading data directly from the AO CTree database. That would of course require someone to port aoppa to Windows. Auno.org, There are several files in the repository that are directly related to my Auno.org website. These include a special PostgreSQL output plugin, the sql schemas and some tools for creating additional tables out of the data in aodb_* tables. Some of them might be interesting for others who want to develop an AO database website, most are probably uninteresting, though. Getting started, Get the latest .tar.gz package from http://auno.org/dev/aoppa/ and gunzip and untar it in your favourite Unix system. Then run 'make' to generate aoppa and the plugins and run: ./aoppa -p input-gzip.so -p output-xml.so <input files> Where the <input files> are your database dumps generated by ADBD. If you don't want or are unable to run ADBD you can also fetch 18.0.0 item and nano dumps from http://auno.org/dev/aoppa/items-18000000ep1.dat.gz and http://auno.org/dev/aoppa/nanos-18000000ep1.dat.gz After running the command you can just watch the XML output scroll by... Development, I have been meaning to release this code for public for as long as I have been working on it, but there's always been one or another thing that has blocked the release. I have wanted to fix some of the most obvious and embarrassing bugs in the code before a release, but the past has proved that the chances are that it'll never happen unless I just go ahead and give out the current version of the code. As you can see, I have chosen the GNU General Public License for the project. What it means is that you can freely copy, modify, sell, and distribute aoppa, but you have to give the full source code with any binaries you ship. This includes the source code for any derivative works, or anything that is linked with Aoppa. This includes all plugins. I would also like anyone who modifies the code to send diffs of their modifications back to me, so I can update my version and post any improvements on my website. If there is a lot of interest in this we can probably also setup a Sourceforge project. Anything and everything else, If you have any questions about this feel free to send me an email at [email protected] or write in the forums at http://auno.org/ Cheers, Auno Bitnykk 2020 Addition (with Assx help) : You may need to install : - at least the build essentials package for the main aoppa processing - also postgresql (& its libpq-dev) for optionnal aunoorg-mk process Once ready, for aoppa just do : make It should compile aoppa and its .so & .o files Example of their use, e.g. to produce an XML file from AO DB extract by adbd : ./aoppa -p input-gzip.so -p output-xml.so adbd-source.dat > aoppa-destination.xml For auno* tools : make -f aunoorg-mk It should compile several usable PG libraries Example of their use, e.g. to connect & update your local PG* DB from AO DB extract by adbd : ./aoppa -p input-gzip.so -p aunoorg-output.so -o dbname=mypgdatabase,dbuser=mypguser,userpass=mypguserpass adbd-source.dat *: for the PG part refer to these first steps https://wiki.debian.org/PostgreSql From there you'll execute everything under mypguser in both command line & DB. Once ready you may need to import auno's aodb structure : psql -U mypguser -d mypgdatabase -f aunoorg-sql.schema psql -U mypguser -d mypgdatabase -f aunoorg-sql.index And then, once .dat was imported into DB, you may want to dump all from PG : pg_dump mypgdatabase > dumpresult.pgsql Or may be dump a specific table only, e.g. for sql insert : pg_dump --column-inserts --data-only --table public.aodb mypgdatabase > tableinsert.sql Or even possibly some specific fields from one or more table(s) to csv, e.g. : psql -hlocalhost -dmypgdatabase -Umypguser -c "copy(SELECT public.aodb.aoid, public.aodb.ql, public.aodb_ext.icon, public.aodb.name FROM public.aodb INNER JOIN public.aodb_ext ON public.aodb.xid = public.aodb_ext.xid WHERE public.aodb.ql > 0 AND public.aodb_ext.icon > 0) to stdout DELIMITER ';'" > mypgexport.csv This could by the way be optimized with Budabot ItemsExtractor filter (delete_list.txt) by Tyrence at https://github.com/Budabot/ItemsExtractor/blob/master/src/main/resources/ And if ever needed, you may want to drop & recreate the DB as earlier : su - postgres dropdb mypgdatabase createdb -O mypguser mypgdatabase Then just repeat all upper imports with schema, index & .dat file(s)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.