Abstract Copy internals more, improve logging, etc

• Firstly, define users table
  • id (int & auto increment & Primary Key (PK))
  • name - not index
  • uuid (string) - index
  • height. ("5ft9" - string)
  • shoesize (number) must support Smol size - not a whole number (4.5)
  • age (integer)
  • bio (string)
  • friends_count (random between 3 - 75)
  • favourite_animal (string)
  • favourite_colour (string)
  • favourite_food (string)
  • mobile_phone (string)

Technical A/C

• One of the tests need to be write 1,000,000 insert queries into MySQL.
• Drop database completely before it does the test suite.

Summary
The purpose of this ticket is to prime an example MySQL db.

The intention is to write this inside the Go package so that eventually it can be in CI/CD.

Maybe use Faker for things like the favourite animal just to ensure that all sorts of string lengths and data is in the DB.

Outcomes

• This could be recycled for other test suites, e.g. running 10,000 records with the replication later.
• Ensure that the use of MySQL driver in the solution can handle this.
• Pick a suitable testing framework (that automatically just profiles the amount of time it does).

Expected Behavior

Performance in CopyToString should be comparable to that of CopyToList and CopyToHash.

Current Behavior

CopyToString is over 100% slower than the aforementioned two.

Possible Solution

Because of Redis' data structures, CopyToString (strings) has to write one key per field, compared to CopyToList and CopyToHash that both write one key per row. This is not necessarily something that can be "quick fixed", but with other performance optimizations, especially transactions and pipelining, performance should improve drastically.

Steps to Reproduce

1. Run test cases with ~10k rows or so and your preferred DB.
2. View output:

➜  redisql git:(master) go test -v -db mysql -rows 10000 
Preparing Test...
=== RUN   TestCopyToString
--- PASS: TestCopyToString (8.43s)
=== RUN   TestCopyToList
--- PASS: TestCopyToList (0.75s)
=== RUN   TestCopyToHash
--- PASS: TestCopyToHash (0.92s)
PASS
ok      github.com/DGKSK8LIFE/redisql   17.995s

Expected Behavior

• The highlighted list item should be removed as I've since removed logging entirely from the library.

Current Behavior

• The highlighted list item is not removed

Ran recent test cases with logging on/off, found disabling logs decreased latency on average about 50% 😬.

More feedback, active users, contributors, etc

Describe the solution you'd like

Provide a docker compose solution so a maintainer or user can run this box in isolation of needing PostgreSQL/MySQL and Redis to test this.

Describe alternatives you've considered

A docker image could be appropriate for testing which combines Go, MySQL, PostgreSQL and/or Redis in one box, however it would require maintenance of upstream Docker images and there's no performance penalty using docker compose, a docker-compose template allows maintainers to just completely retrash/rebuild their development environment by services.

How features should be contributed from now on:

Feature branch 👉🏻 Unstable 👉🏻 Included in stable release when ready

Give better config options in terms of formatting, pretty-printing, log-file support, etc.

Could be implemented in a redis-native way with Redis pub/sub; not sure about MySQL

(Would follow redis naming conventions and ideally give options for override)

• Have different levels of logging
• Make logging have little impact on performance

Currently hash names are generated with a UUID, but I want to preserve SQL table names, so I will use the Redis naming convention (object-type:id). Also want to give developers the option to override this.

Would do this either by passing a struct with one flag or using a config file

A new major release will fix all if not most current issues, optimize the library, introduce major refactors, etc overall.