It has happened to me several times now that I ingest a large set of repositories, I l

The authors table gets derived from <code class="notr

a workflow to remove a repository after ingestion? about fornalder HOT 1 OPEN

hpjansson commented on July 17, 2024

a workflow to remove a repository after ingestion?

from fornalder.

Comments (1)

hpjansson commented on July 17, 2024

The authors table gets derived from raw_commits every run, so it should be safe to poke around in the latter. See:

fornalder/src/commitdb.rs

Lines 204 to 233 in 43f3d48

 // Generate table with per-author stats like time of first and 

 // last commit. 

 self.conn.execute ("drop table authors;", NO_PARAMS).ok(); 

 self.conn.execute (" 

  create table authors as 

  select author_name, 

  first_time, 

  first_year, 

  last_time, 

  last_year, 

  last_time-first_time as active_time, 

  n_commits, 

  n_changes 

  from 

  ( 

  select author_name, 

  min(author_time) as first_time, 

  min(author_year) as first_year, 

  max(author_time) as last_time, 

  max(author_year) as last_year, 

  count(id) as n_commits, 

  sum(n_insertions) + sum(n_deletions) as n_changes 

  from raw_commits 

  group by author_name 

  ); 

  create index index_author_name on authors (author_name); 

  create index index_first_time on authors (first_time); 

  create index index_active_time on authors (active_time); 

  ", NO_PARAMS).chain_err(|| "Could not create author summaries")?;

I intended to re-run postprocess() only if something changed (e.g. store a hash of the meta file provided, clear a flag whenever a fornalder command like ingest changes the database), but it wasn't too slow in practice, so I didn't feel the need to optimize it, at least not yet. I left a reminder here:

fornalder/src/main.rs

Line 217 in 43f3d48

cdb.postprocess(&meta.domains)?; // FIXME: Skip if metadata is unchanged

Anyway, the bottom line is that manually editing raw_commits is safe, for now.

I like the idea of having CLI for common database editing (like removing a repo, or maybe a date range). Let's keep this issue open for repo-remove (or remove-repo?).

from fornalder.

Recommend Projects

a workflow to remove a repository after ingestion? about fornalder HOT 1 OPEN

Comments (1)

Related Issues (9)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	// Generate table with per-author stats like time of first and
	// last commit.

	self.conn.execute ("drop table authors;", NO_PARAMS).ok();
	self.conn.execute ("
	create table authors as
	select author_name,
	first_time,
	first_year,
	last_time,
	last_year,
	last_time-first_time as active_time,
	n_commits,
	n_changes
	from
	(
	select author_name,
	min(author_time) as first_time,
	min(author_year) as first_year,
	max(author_time) as last_time,
	max(author_year) as last_year,
	count(id) as n_commits,
	sum(n_insertions) + sum(n_deletions) as n_changes
	from raw_commits
	group by author_name
	);
	create index index_author_name on authors (author_name);
	create index index_first_time on authors (first_time);
	create index index_active_time on authors (active_time);
	", NO_PARAMS).chain_err(\|\| "Could not create author summaries")?;