Disclaimer: As I mention below, I built this application with scale in mind, but I want to make clear that this app is not intended for further development and user adoption. The only files I have to work with are images which are the intellectual property of various comics publishers and creators, and the last thing I want it to build a tool for easily pirating from a fragile industry that means a lot to me. For the same reason, and because the purpose of this project was to learn about back end technologies the front end client that exists is meant only to serve the barest needs of the system, ie uploading sets of image files, querying for previously stored comic book issues, and viewing those images in their full resolution. There is a bit of test content in the database, some recent DC comics, and if you would like to see the app in action please feel free to search for the year 2019 to see all the issues currently in the database, and click the read button on the cards that show up. If you work for DC or are otherwise concerned about my making this content available here, please let me know and I will remove it immediately. This was for educational purposes only.

This digital comcics reader application was a tightly time-boxed (2 days) experiment in hosting and serving larger image files without the benefit of hosted solutions like AWS S3. The first major challenge in achieving this goal was deciding on a method for actually storing the images on the server. The straightforward option would be to use normal file storage, and manage queries by matching them to filepaths. This would totally have been doable, but it would introduce challenges for associating metadata with the comics issues and make it more difficult to build out navigation between related issues or with inexact queries. On the other hand I knew that database management systems (DBMS) like PostgreSQL have been offering image storage and, more importantly, binary large object (blob) storage for many years, and I was curious to know more about these tools. I read plenty of kneejerk opinion online that this wasn't really the sort of thing relation DBMS were built to do, and maybe that's true, but it seemed strange to me that PostgreSQL would continue to support such a feature if it just didn't really work and if nobody was using it. I did a little more digging and found some good research benchmarking the performance of blob storage and service in differnt database systems. It turns out that as long as the blobs are under 5mb PostgreSQL's large object storage performs very well. That's not a huge file, admittedly, but the standard size of a digital comics page is between 1mb and 4mb, so PG's LOb storage is perfect for this sort of thing.

The other major challenge was figuring out how to actually get the image files from the user interacting through the browser, through the server, and into the large object storage. While the size of the data at hand isn't huge, a comic book has 20-30 pages, multiplied by about 3mb per page, the rseulting ~75mb per upload is more than I want my server to have to buffer in memory at once. I could certainly make it do that, but my server is an AWS EC2 t2.micro, so memory is precious, and I like to build anything with ability to scale from the outset. If very many people at all were using an app like this, buffering all those images into memory would be a nightmare for the server. Multipart form data would handle the front end needs of the application in any case, but for the server and database adapter I ultimately decided on using Node streams passing through a very lightweight Connect server (think Express with the shiny chasis ripped off) into Postgres transactions in order to ensure data integrity on lost connections. That is, with a series of transactions instead of simple/batch queries, if there is a hiccup in the connection the whole series of transactions stops instead of getting partial data into the database which would be a major storage problem.

Having never worked directly with streams before, it was a challenge to comprehend them and get them working in production in just one day, but I have found ultimately that they are a powerful and intuitive tool. I have learned that they were one of the big selling points for Node.JS early on and I can see why.