@00-matt pointed out that uint32_t should be used instead of int when storing integers in the fist.db file. Since the size of int isn't standard on some systems the int could be 2 bytes instead of the currently assumed 4 bytes causing the db file to become corrupted.

So this needs to be fixed.

Not sure if I would call this a bug, but when the index gets very large and the server is closed the sdump function takes a very long time causing the server to look like its hanging.

A quick improvement would be to create a message stating that Fist is attempting to Exit cleanly before sdump is run, but there is a deeper problem to be solved here so that's why I am opening this issue.

Hi,

This might be a stupid way to ask a question on Github but I want to ask you this question. Why did you write it in C and not in C++?

Also I would like to contribute to it. How can I?

Creating this issue to remind myself that it is an issue and also to create some conversation around this.

The current state of the server only allows for a single client to be connected at a time. So this means that if there are multiple applications that need to access the same Fist instance, one client will have to wait for the other to disconnect before it can start sending commands.

This was just easier to do at first but it will need to be addressed because it's not acceptable.

With the addition of a configuration file and a few new commands it's time to properly document everything.

Indexing algorithm can (should) be able to be sped up by using a divide and conquer algorithm.

There is currently no way to delete an entry into the database. A delete command needs to be implemented.

This issues was discussed in more detail here:

#23 (comment)

While attempting to index a lot of Documents, I also tried to send a few SEARCH commands to see how things would behave. While the indexing was still going on, SEARCH commands became very very slow.

I've enjoyed reading this source code. Thank you for sharing. I also understand that it is still in development so my question is really, did I miss something or are my expectations to be met with a future modification?

I see max_phrase_lenght is set to 10.

Is it expected that I could index a document longer than 10 words? I would expect it to index all phrases present in the document up to 10 words long. 10 would seem a generous number in this case. Maybe 3 would be sufficient?

A query then might search for all phrases up to max_phrase_length, favoring the longest matches. Again, with this behavior, I would expect 3 to offer good results.

I would expect dump and load to require consistent values of max_phrase_length. Or do you anticipate accommodating changes somehow?

Allow the user to specify stop words to be not indexed, like the and is.

Is the project continuing?

Currently we are using a very primitive hashing algorithm for the hashmap. A better algorithm should be used to help better prevent collisions.

Would you like to add more error handling for return values from functions like the following?

• malloc ⇒ dappendc
• printf ⇒ handle_connection

This is a bug that is also in the release version 0.0.1 and might have something to do with PR
#15

The below script pulls the Beemovie transcript from Pastebin and attempts to index it. When run, the Fist server stops suddenly with no error message.

I used this code when I was originally testing indexing large files and it worked file (Other than issue #18)

import socket
import requests
import string

data = requests.get("https://pastebin.com/raw/Gb02THWc").content.decode().replace("\n", ' ').replace('\r', ' ').replace('  ', ' ').replace("  ", " ").lower()
for p in string.punctuation:
    data = data.replace(p, '')

data = ' '.join(list(filter(None, data.split())))
print(data)

s = socket.socket()
s.connect(("localhost", 5575))
s.send("INDEX beemovie {}\r\n".format(data).encode())
print(s.recv(1024))
s.send(b"SEARCH bee movie\r\n")
print(s.recv(1024))
s.send(b"EXIT\r\n")

When indexing large blocks of text (e.g. a movie script) not all phrases get indexed properly. It seems like randomly some phrases/words get indexed while others do not.

• setup sass
• choose fonts
• find favicon
• add code highlighter
• add nav links to github, patreon, slack
• style footer
• add social media icons
• add links to clients
• fix mobile version
• add media queries

Need to write tests for the binary search tree implementation in bst.c and bst.h

Currently the index is stored as is in the fist.db file. As the index grows it will require a lot of space. Some kind of compression will need to be implemented to allow the index to get sufficiently large without consuming a huge amount of resources.

EDIT: As an example of how important this is, I attempted to index all of the Joe Rogan podcast transcripts found here( https://github.com/achendrick/jrescribe-transcripts ) The index grew to be over 1.5gb. When compressed (manually) with gzip the file size was brought down to 300mb.

like matt said #50 , need a stop words, or a limit output, or rank option(need INDEX support)

Hi,

It would be great to get a version number going so that we could track changes that break compatibility with the protocol or database file.

Settings such as max_phrase_length, host, port, and buffer_size` should be configurable via a config file.

The signature of calloc looks like this:
void* calloc (size_t num, size_t size);
In the current hashmap allocation, num of elements depends on the size of the structure, which doesn't seem correct.