hi-primus / bumblebee Goto Github PK

See the gif
https://recordit.co/Ek4Y6CCrf3

None of the users interviewed use the columns operations search, sort, filter. We can reconsider to move this to the right side of the toolbox.

For date datatypes, we should create an easy way for the user to select the format they want to match. There are hundreds of combinations between days. months, years, minutes, seconds. @luis11011 any idea how can we present this to the users?

Load file

• Rename "Upload" for "Preview"
• Move "limit" inside "infer"
• Rename "infer" to "More options"
• Move the url load option to "More options"

Operations

• no case senstive in the filter operation. Use ignore_case param in replace function. See hi-primus/optimus@f3702ce
• If an operation fails do not add it to the notebook
• The columns search box is not working. Is not filtering any column

Right now, Bumblebee receives data via sockets from Optimus. This cause problem because all the sampled data must be loaded into the browser hitting hard in the performance. Using MongoDB we can read only the rows needed to render the table.

Optimus should send the profiled data to MongoDB.
Bumblebee must load only the data needed to show in the table.

It would be great to create a Dockerfile so user can easily install bumblebee.
Here the installation instrucctions https://blog.hi-bumblebee.com/install-bumblebee/

Right now when loading data from to the data window we load only the visible rows. We need to make this for columns too. In this way, we can reduce the latency and increment the performance.

Hi I think we should be posting some visual information in the Readme so people have an idea of what it looks now, some screenshots or gifs would work.

Let me know your thoughts on this @argenisleon @JorgeParraAndrade

It's necessary to receive the profiler data from a queue. In this way, we can receive data from any source from local installations, on-premise or from a third party like databricks.

The user should be able to get a file preview to select the best params like sep or charset to get the output required. We can do it easily using the n_rows param when loading the data

We should teach the user the basic Bumblebee functionality through a live onboarding.
We could use de foo.csv file in /example/data/
Some basic function we can show are:

• Load Data
• Clean accents and special chars
• Unnest
• Nest
• Filter
• String clustering
• Outliers
• Save data to a file

I tried opening the link https://blog.hi-bumblebee.com/install-bumblebee/ but it shows connection has timed out.

This is a reference for google docs

Use the new added Optimus param ignore_case=True to ignore case when searching for replacing

The Optimus now detect these datatypes:

• INT
• DECIMAL
• STRING
• BOOLEAN
• DATE
• ARRAY
• OBJECT
• GENDER
• IP
• URL
• EMAIL
• CREDIT_CARD_NUMBER
• ZIP_CODE
• CATEGORICAL

they are sent in the profiler information an can be found in the profiler_dtype key

{'columns': {'INCIDENT_NUMBER': {'stats': {'mismatch': 0,
    'missing': 0,
    'match': 319073,
    'profiler_dtype': 'string',

Right now Optimus handle fingerprints, ngram-fingerprint, Levenstein distance to cluster strings. Using the string grouper library could give a boost to the clustering Related to hi-primus/optimus#919

We are using Jupyter Kernel Gateway, but the project seems buggy(WebSocket echo and random disconnections) and no longer maintained.
We should migrate to https://github.com/jupyter/enterprise_gateway

The user can already load data from a different source but he can not be easily handled or merged. We can make sure some UI/UX improvements to make it work.

Also seems to be adding some kind of HTML element at the end of the string

These functions have been added to Optimus development branch.
def round(self, input_cols, decimals, output_cols=None):
def floor(self, input_cols, output_cols=None):
def ceil(self, input_cols, output_cols=None):

@luis11011 here are some of the joining features:

• You user can make join operation with any tab opened in the workspace.
• Show the key icon only on hover. Like the visibility button in the list view for column selection.
• I am exploring how can we visually show the user the two data set some work. I am trying light mixing https://www.figma.com/file/OaOQviw4uzfpHl7Uwn8uyK/Join-color-scheme-test?node-id=0%3A1. I little bright right now nut is a start.

This can be useful to get an overview of string structure of a columns
def patterns(self, input_cols, output_cols=None, mode=0):

See https://github.com/ironmussa/Optimus/blob/develop-3.0/optimus/engines/base/columns.py#L153 For more info about the param

Let users create accounts using and email

Users should be able to upload files and in csv, json, parquet, avro

I think this would open a new tab and will try to preview the file. The user would need to confirm the operation to finish the data loading.

• If there are duplicated values show only the rows that match
• Show how many duplicated values were found
• Make clear to the user which columns are been operated
• If not matches were found give feedback to the user

@luis11011 Seems there are two extra span elements every cell. Could we get rid of this? I think this could be hitting the performance.

Optimus now accepts for pandas an dask:

new_col = "std_col"
new_col_1 = "min_col"
new_col_2 = "count_col"
_agg = {new_col:{"Quantity":"std"},new_col_1:{"Quantity":"min"},new_col_2:{"ProdCategory":"count"}}

df.cols.groupby(by="State", agg = _agg).rows.sort("count_col", "desc").ext.display()

The aggregations supported are:

• count
• sum
• mean
• std
• min
• max
• first
• last

@luis11011 The user should be able to select a column and add multiple aggregations; naming the output column, the column he wants to aggregate and the aggregation

These are the operations that highlight rows:

Toolbar

• Drop duplicates
• Drop nulls
• Filter

Charts

• Select bars in chars
• Select bar in frequency

Quality bar

• Missings
• Mismatches

• If there are match show only the rows that match
• Make clear to the user which columns are been operated
• Show how many matches were found
• If not matches were found give feedback to the user

Right now Bumblebee can save data to files, It will be useful if the user can save data to databases. Optimus already have these features so we can implement them easily.

After making a full operation over the whole dataset, we could make a realtime preview.

We could start with:

• Replace
• Nest
• Unnest
• Filter
• Rename
• Duplicate

There is an additional pixel between the head and the data in the column.

After data loading, we should show a profiling screen with the main data stats info useful to the user.

Menu "add data source"

• Add from file
• Add from database
• Add from a loaded data source
• Manage data sources

Menu "save"

• Download
• Save to database

UX

• Add data source call to action when opening a new tab
• Open the load dataset dialog when opening a new tab
• Show more recent data sources
• Drag and drop from file

The user should be able to drop rows with nan values
In dask:
df.rows.drop_na(how="all").compute()

Options for the user

• all
• any

Preview for pandas
pdf["__match__"] = pdf.isnull().any(axis=1)

We are going to use vega-lite https://vega.github.io/vega-lite/.

The notebook view seems overkill for most users. A plain English description could be enough for the user to have an overview of all the operations.

@luis11011 as we talk some weeks ago we could just pick the operation and let the user modify the params

Put these functions under an ML icon in the toolbar

• Sampling
• Prepare
• Impute
• Standard
• Min Max
• Max abs
• Values to Columns
• String to Index
• Indices to strings
• Outliers

The user should be able to drop duplicates rows.

options

keep

• firsts
• last

Code for preview

pdf["__match"] = pdf.duplicated(keep='')

Code for dask

df.rows.drop_duplicates(keep='').compute()