cmu-db / bustub Goto Github PK

View Code? Open in Web Editor NEW

3.7K 41.0 1.7K 8.82 MB

The BusTub Relational Database Management System (Educational)

Home Page: https://15445.courses.cs.cmu.edu/

License: MIT License

CMake 2.58% Python 1.87% C++ 94.30% Shell 0.30% HTML 0.90% C 0.05%

database dbms

bustub's People

Contributors

Stargazers

Watchers

Forkers

lmwnshn mbutrovich serrislew diegomestre2 moonjoaws zhengzheyang wenhao-ma songzhaozhe ziyi-yan dingqy wqlin barneybaby strrl tangwz wty4427300 kgtdbx laoawilliam richard-chau xinzhu-cai snithish andrezwei zz-jason akalsi87 tupaschoal alala39 schneiderlin fcofdez pdaviso shengruilyu sxnan jhan27 bcli-1999 jennybkim alkaxearathu himanshu141 donghaowen zhou0ricky apavlo cinashi gengyixuan gengkev whalecold rsy56640 js8544 wuli749826 thepulkitagarwal rgwood xiaoxlu wenxuanqiu vaibhav-sinha simeng96 jiaqiliu gindachen skyshaw huutfj alexbyrnes3 jssprasanna mmoskal98 favour-dgreat yixuaz allenclown shuheiktgw riflerrick abdulrahman86 nikhil1008 vctrkang tcalasans sudk1896 franceshe 0x584c utkarsh39 deyuhua xgliu johncoconut bakircikagan anuranjit geauxeric yvxiang ivanallen shichao-wang mabondano chungjin yuxineverforever kevinjzhang chrisshyi kyan666 zhong-zh13 stonewhitener andryleon rahulsmahadev harsha3294 axispx 12ds95 debnil kirugan astronaut0131 chihiro2014 bunnybrewery chaogu everac

bustub's Issues

Cannot run command "sudo ./build_support/packages.sh" on AWS EC2 and shark machine

I am using AWS EC2 t2.micro instance
When I run the command sudo ./build_support/packages.sh according to cmu-db/bustub
The Linux console prints out:

ubuntu@ip-172-31-81-17:~/15-645/private-bustub-master$ sudo ./build_support/packages.sh
PACKAGES WILL BE INSTALLED. THIS MAY BREAK YOUR EXISTING TOOLCHAIN.
YOU ACCEPT ALL RESPONSIBILITY BY PROCEEDING.
Proceed? [Y/n] : Y
++ tr '[:lower:]' '[:upper:]'
++ uname
+ UNAME=LINUX
+ case $UNAME in
++ cut -d '"' -f 2
++ grep VERSION_ID
++ cat /etc/os-release
+ version=16.04
+ case $version in
+ give_up
+ set +x
Unsupported distribution 'LINUX'
Please contact our support team for additional help.
Be sure to include the contents of this message.
Platform: Linux ip-172-31-81-17 4.4.0-1104-aws #115-Ubuntu SMP Mon Mar 2 06:35:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

https://github.com/cmu-db/bustub/issues

Also, I tried to run on shark machine (without using sudo), it prints out:

-bash-4.2$ ./build_support/packages.sh
PACKAGES WILL BE INSTALLED. THIS MAY BREAK YOUR EXISTING TOOLCHAIN.
YOU ACCEPT ALL RESPONSIBILITY BY PROCEEDING.
Proceed? [Y/n] : Y
++ uname
++ tr '[:lower:]' '[:upper:]'
+ UNAME=LINUX
+ case $UNAME in
++ cat /etc/os-release
++ grep VERSION_ID
++ cut -d '"' -f 2
+ version=7.8
+ case $version in
+ give_up
+ set +x
Unsupported distribution 'LINUX'
Please contact our support team for additional help.
Be sure to include the contents of this message.
Platform: Linux angelshark.ics.cs.cmu.edu 3.10.0-1127.18.2.el7.x86_64 #1 SMP Mon Jul 20 22:32:16 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

https://github.com/cmu-db/bustub/issues

buffer_pool_manager_test has a minor problem

bustub/test/buffer/buffer_pool_manager_test.cpp

Line 37 in caf0869

snprintf(page0->GetData(), sizeof(page0->GetData()), "Hello");

sizeof(page0->GetData()) is 8 bytes on a 64-bit OS or 4 bytes on a 32-bit OS.

Possible compile error, if the following string having size larger than 8.
Example with "Hello World":

/tmp/bustub/test/buffer/buffer_pool_manager_test.cpp:22:6: error: ‘Hello World’ directive output truncated writing 11 bytes into a region of size 8 [-Werror=format-truncation=]

The buffer here is the page, so the buffer size is PAGE_SIZE.

Possible relevant: How to find the 'sizeof' (a pointer pointing to an array)? - Stackoverflow

When can the full test cases be publicly available?

Just want to judge the correctness of the project implementations more comprehensively.🙂

How to log information with generic_key

I'm trying to debug project 2, but don't know how to log KeyType, ValueType i.e. GenericKey... using LOG_INFO.
There are 2 functions In "generic_key.h"


  // NOTE: for test purpose only
  // interpret the first 8 bytes as int64_t from data vector
  inline int64_t ToString() const { return *reinterpret_cast<int64_t *>(const_cast<char *>(data_)); }

  // NOTE: for test purpose only
  // interpret the first 8 bytes as int64_t from data vector
  friend std::ostream &operator<<(std::ostream &os, const GenericKey &key) {
    os << key.ToString();
    return os;
  }

but how to use it with LOG_INFO/LOG_ERROR ?

BufferPoolManager.NewPage spec is unclear if the pages returned must be pinned

It is unclear from the spec if the pages returned from BufferPoolManager.NewPage must be pinned.

Actually, if they are not the test would fail (as any previously allocated page becomes victimized instantly), from what I can conclude they must be?

And of course, it is funny to return a pointer to a newly allocated page which was concurrently evicted even before the method returned.

It is also unclear whether the new page must be marked as dirty.

check-lint assumes python is symlinked to python2

If this happens to you

$ make check-lint
Scanning dependencies of target check-lint
make[3]: *** [CMakeFiles/check-lint] Error 1
make[2]: *** [CMakeFiles/check-lint.dir/all] Error 2
make[1]: *** [CMakeFiles/check-lint.dir/rule] Error 2
make: *** [check-lint] Error 2

i.e. check-lint doesn't seem to show you any errors, the reason is because Google's cpplint assumes that python will run python2. More specifically, the python first found in your PATH must be python2.

The workaround is to run PATH=/usr/bin:$PATH make check-lint. On most operating systems with python symlinked to python2, this should work. If you're running Arch Linux or something else with python symlinked to python3, you should know how to fix it yourself.

I've been meaning to convert their script to work with Python 3, but I don't have the time. A good first step is probably running 2to3 on it.

If you fix this issue, your fix will be ported to terrier which faces the same problem. I think Apache Arrow might have similar cpplint issues too.

Project 1 : make format failed

➜ build git:(master) ✗ make format
Traceback (most recent call last):
File "../build_support/run_clang_format.py", line 125, in
had_err = check(args, source_dir)
File "../build_support/run_clang_format.py", line 51, in check
"-i"] + formatted_filenames)
File "/usr/lib/python2.7/subprocess.py", line 185, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 172, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 394, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
make[3]: *** [CMakeFiles/format.dir/build.make:57：CMakeFiles/format] 错误 1
make[2]: *** [CMakeFiles/Makefile2:280：CMakeFiles/format.dir/all] 错误 2
make[1]: *** [CMakeFiles/Makefile2:287：CMakeFiles/format.dir/rule] 错误 2
make: *** [Makefile:214：format] 错误 2

clang-tidy and clang-format finder

Figure out what changed in recent versions of Mac / homebrew; where are the clang-format and clang-tidy binaries being installed? Works for some people, doesn't work for others. I don't have a recent Mac, so I'm probably going to rely on someone else to fix this.

Temporary workaround: figure out where clang-tidy and clang-format are, and add them to your PATH.

https://github.com/cmu-db/bustub/blob/master/CMakeLists.txt#L15

Will solutions of the whole project be available to the public someday?

Bustub is really outstanding course material for practice. Though I had finished all four projects, they could not pass all the tests in Gradescope after a one-month-long debug still. So could all solutions be released someday please? Just for learning. I'd appreciate it if so.

Turn on naming conventions checker

Right now, we're not enforcing terrier-style naming checks on the codebase. We didn't switch this check on early enough, and we don't want to make students go back and fix up their old code only for style. So we'll wait until the semester is over before we enable this check.

Just uncomment these lines.

Add page_type enum to every Page

We want a page_type enum in every Page class. Useful for autograding.

https://github.com/cmu-db/bustub/blob/master/src/include/storage/page/page.h#L28

Can we have this project in Java please at some point?

how to understand the second test case of buffer_pool_manager_test.cpp?

TEST(BufferPoolManagerTest, DISABLED_SampleTest) {
  const std::string db_name = "test.db";
  const size_t buffer_pool_size = 10;

  auto *disk_manager = new DiskManager(db_name);
  auto *bpm = new BufferPoolManager(buffer_pool_size, disk_manager);

  page_id_t page_id_temp;
  auto *page0 = bpm->NewPage(&page_id_temp);

  // Scenario: The buffer pool is empty. We should be able to create a new page.
  ASSERT_NE(nullptr, page0);
  EXPECT_EQ(0, page_id_temp);

  // Scenario: Once we have a page, we should be able to read and write content.
  snprintf(page0->GetData(), PAGE_SIZE, "Hello");
  EXPECT_EQ(0, strcmp(page0->GetData(), "Hello"));

  // Scenario: We should be able to create new pages until we fill up the buffer pool.
  for (size_t i = 1; i < buffer_pool_size; ++i) {
    EXPECT_NE(nullptr, bpm->NewPage(&page_id_temp));
  }

  // Scenario: Once the buffer pool is full, we should not be able to create any new pages.
  for (size_t i = buffer_pool_size; i < buffer_pool_size * 2; ++i) {
    EXPECT_EQ(nullptr, bpm->NewPage(&page_id_temp));
  }

  // Scenario: After unpinning pages {0, 1, 2, 3, 4} and pinning another 4 new pages,
  // there would still be one buffer page left for reading page 0.
  for (int i = 0; i < 5; ++i) {
    EXPECT_EQ(true, bpm->UnpinPage(i, true));
  }
  for (int i = 0; i < 4; ++i) {
    EXPECT_NE(nullptr, bpm->NewPage(&page_id_temp));
  }

  // Scenario: We should be able to fetch the data we wrote a while ago.
  page0 = bpm->FetchPage(0);
  EXPECT_EQ(0, strcmp(page0->GetData(), "Hello"));

  // Scenario: If we unpin page 0 and then make a new page, all the buffer pages should
  // now be pinned. Fetching page 0 should fail.
  EXPECT_EQ(true, bpm->UnpinPage(0, true));
  EXPECT_NE(nullptr, bpm->NewPage(&page_id_temp));
  EXPECT_EQ(nullptr, bpm->FetchPage(0));

  // Shutdown the disk manager and remove the temporary file we created.
  disk_manager->ShutDown();
  remove("test.db");

  delete bpm;
  delete disk_manager;
}

Project 2 cannot pass gradescope test due to internal header problems.

test/include/logging/common.h:36:16: error: the const qualified variable 'col' is copy-constructed from a const reference; consider making it a const reference [performance-unnecessary-copy-initialization,-warnings-as-errors]
    const auto col = schema->GetColumn(i);
               ^
              &

When I was trying to grade Project 2 on gradescope, I found that check-lint cannot succeed due to the error above. There is a historical commit on bustub which fixed this problem. So it seems the code on gradescope is the earliest version. Can you please update the internal code on gradescope? Thanks!

Aggregation Executor - How to get correct ordering of columns in output tuple

In Project 3 Task 2, for the aggregation executor, I can't see how to ensure the Next() method obeys the OutputSchema column ordering. I have an AggregateKey and an AggregateValue, but I don't see how I can know in which order I should combine their contents to create the correct output tuple, given that they are both essentially vector types and as far as I can see the Value type gives no information about which column it corresponds to. I feel like I'm missing something obvious.

As an example, in ExecutorTest - SimpleGroupByAggregation, the provided output schema has 3 columns: countA, colB, sumC where colB is a group by column. I can see how we can create a tuple that has all of the group by columns to the left, and all aggregates to the right:

AggregateKey key = aht_iterator_.Key();
AggregateValue vals = aht_iterator_.Val();
const size_t num_cols = key.group_bys_.size() + vals.aggregates_.size();
assert(num_cols == plan_->OutputSchema()->GetColumnCount());
std::vector<Value> combined_cols;
combined_cols.reserve(key.group_bys_.size() + vals.aggregates_.size());
combined_cols.insert(combined_cols.end(), key.group_bys_.begin(), key.group_bys_.end());
combined_cols.insert(combined_cols.end(), vals.aggregates_.begin(), vals.aggregates_.end());
*tuple = Tuple(combined_cols, plan_->OutputSchema());

I understand that the output schema specifies the ordering we want, but I don't see how it is possible to know which values in the AggregateKey and AggregateValue correspond to which columns in the output schema, and therefore how to combine them to obey the output schema.

Any help would be greatly appreciated. Thanks in advance.

allow flushing pages with read only latch

In order to allow flushing a page to the disk with a weaker mode (read lock) than exclusive lock (write) a page needs either:

atomic dirty flag
atomic page modification version
RWLatch to support rising latch level (rising read lock to write lock)

It also can be done with a separate structure of flushing reservations.

Why does HashTableBlockPage::Insert need to be thread-safe?

bustub/src/include/storage/page/hash_table_block_page.h

Line 60 in 7704f90

* The insert is thread safe. It uses compare and swap to claim the index,

The comment suggests we use CAS here to make insert thread safe. However, from task #4 description,

You will need to have latches on each block so that when one thread is writing to a block other threads are not reading or modifying that index as well.

I'm wondering why we still need CAS when latches ensure only one thread is modifying the block page?

Actually, if we use CAS to make sure HashTableBlockPage::Insert and HashTableBlockPage ::Remove are thread-safe, I think we don't need page latches. The table latch is enough. Am I missing something here?

Last but not least, it would be really helpful if #58 can be fixed. 😄 Looks like tests and /src are out of sync so the grading script wouldn't work. Now I run multiple threads to insert(resize) and delete concurrently for testing.

LRU Testing : lru_replacer_test.cpp

Line 26, 32, and 38: 1 is unpinned 2 times. Due to the last unpin, 1 becomes the most recently used block and hence should not be evicted in line 38.

buffer_pool_manager_test.cpp : stcrncmp, strcmp

Line 48, 49: It may be a good idea to use memcpy and memcmp for copying and comparing binary data. String functions do not copy beyond the null character.

How to make sure the page id returned by redo newpage is the same as the one in log record?

Currently, for redo newpage, we use the page id in the log record. When we create new pages in recovery, the returned page ids should be the same as the original pages since the disk manager just uses an increasing counter for the page id. The counter starts over when the database is deleted and you're creating new pages is the same order as they were originally created, so the page ids should be the same.

However, what if we want to pass a page id as a parameter to newpage? Seems like we need to add an input parameter to NewPageImpl in buffer_pool_manager. Inside this function, we also need to check whether the requested page id has already been allocated.

Replacer API leading to student confusion

The buffer pool manager has Pin() and Unpin() functions. This is standard terminology, and we should probably keep it that way.

In fall 2019, we renamed the replacer API to have Pin() and Unpin() functions. This led to some student confusion; they were mixing up pinning and unpinning of replacer frames and buffer pool pages.

We should think about better names or a better API for the replacer.

Duplicate file open mode (bit or operation)

bustub/src/storage/disk/disk_manager.cpp

Line 51 in 0bb0669

 db_io_.open(db_file, std::ios::binary | std::ios::in | std::ios::out | std::ios::out); 

I suppose there should be db_io_.open(db_file, std::ios::binary | std::ios::in | std::ios::out);, to open a file, if it exists.

std::ios::out | std::ios::out is not bug (and would be optimized by the compiler), but not readable. 😸

make fails with various compile errors

I am following the steps on README to build the project. When I run make I get the following error

<project_root_dir>/src/include/storage/page/page.h:70:39: error: expected ‘,’ before ‘)’ token static_assert(sizeof(page_id_t) == 4)

Also get
<project_root_dir>/src/include/storage/page/page.h:70:39: error: expected string-literal before ‘)’ token

And

/src/include/storage/page/page.h:71:35: error: expected ‘,’ before ‘)’ token static_assert(sizeof(lsn_t) == 4);

When I comment out the above two lines and try to make then there are a whole bunch other compile errors that I get, some of which are very similar.

'make check-tests' fails due to gtest dependency

I am new to CMAKE. I have just set up the repo, successfully run package.sh, and built the project. However, making tests fail. For the record, I have not written a single line of code thus far.

I run:

cd build
make check-tests

And get

/usr/bin/ld: ../lib/libgtest.a(gtest-all.cc.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: ../lib/libgmock_main.a(gmock_main.cc.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: ../lib/libgmock.a(gmock-all.cc.o): relocation R_X86_64_32 against `.rodata' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
test/CMakeFiles/tuple_test.dir/build.make:89: recipe for target 'test/tuple_test' failed
make[3]: *** [test/tuple_test] Error 1
CMakeFiles/Makefile2:881: recipe for target 'test/CMakeFiles/tuple_test.dir/all' failed
make[2]: *** [test/CMakeFiles/tuple_test.dir/all] Error 2
CMakeFiles/Makefile2:702: recipe for target 'test/CMakeFiles/check-tests.dir/rule' failed
make[1]: *** [test/CMakeFiles/check-tests.dir/rule] Error 2
Makefile:383: recipe for target 'check-tests' failed
make: *** [check-tests] Error 2

Any ideas?
Thank you.

Question about function MoveHalfTo and MoveAllTo

bplus_leaf_page and bplus_internal use different signatures for functions MoveHalfTo and MoveAllTo:

void B_PLUS_TREE_LEAF_PAGE_TYPE::MoveHalfTo(BPlusTreeLeafPage *recipient);
void B_PLUS_TREE_LEAF_PAGE_TYPE::MoveAllTo(BPlusTreeLeafPage *recipient) ;
void B_PLUS_TREE_INTERNAL_PAGE_TYPE::MoveHalfTo(BPlusTreeInternalPage *recipient,BufferPoolManager *buffer_pool_manager) ;
void B_PLUS_TREE_INTERNAL_PAGE_TYPE::MoveAllTo(BPlusTreeInternalPage *recipient, const KeyType &middle_key,  BufferPoolManager *buffer_pool_manager) ;

so I am confused about how template parameter is used in function Split and Coalesce.

Project 1 gradescope fails - cannot find submission files

When submitting the project 1 files to Gradescope, the build technically passes according to Gradescope, but all other tests fail. When I look at the build output (copied below), the build actually fails because it cannot cp the submitted files. Here is the build output:

Cannot find src/include/buffer/clock_replacer.h in submission
Cannot find src/buffer/clock_replacer.cpp in submission
Cannot find src/include/buffer/buffer_pool_manager.h in submission
Cannot find src/buffer/buffer_pool_manager.cpp in submission

cp: cannot stat '/autograder/submission/src/include/buffer/clock_replacer.h': No such file or directory

called process exited with 1
cp: cannot stat '/autograder/submission/src/buffer/clock_replacer.cpp': No such file or directory

called process exited with 1
cp: cannot stat '/autograder/submission/src/include/buffer/buffer_pool_manager.h': No such file or directory

called process exited with 1
cp: cannot stat '/autograder/submission/src/buffer/buffer_pool_manager.cpp': No such file or directory

called process exited with 1

-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- BusTub/main found clang-format at /usr/bin/clang-format-8
-- BusTub/main found clang-tidy at /usr/bin/clang-tidy-8
-- BusTub/main found cpplint at /autograder/submission/project-handout/build_support/cpplint.py
-- Configuring done
-- Generating done
-- Build files have been written to: /autograder/submission/project-handout/build/googletest-download
Scanning dependencies of target googletest
[ 11%] Creating directories for 'googletest'
[ 22%] Performing download step (git clone) for 'googletest'
Cloning into 'googletest-src'...
Already on 'master'
Your branch is up to date with 'origin/master'.
[ 33%] No patch step for 'googletest'
[ 44%] Performing update step for 'googletest'
Current branch master is up to date.
[ 55%] No configure step for 'googletest'
[ 66%] No build step for 'googletest'
[ 77%] No install step for 'googletest'
[ 88%] No test step for 'googletest'
[100%] Completed 'googletest'
[100%] Built target googletest
-- Found PythonInterp: /usr/bin/python (found version "2.7.15") 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- CMAKE_CXX_FLAGS:  -D__BUSTUBFILE__='"$(subst /autograder/submission/project-handout/,,$(abspath $<))"' -fPIC -Wall -Wextra -Werror -march=native -Wno-unused-parameter -Wno-attributes
-- CMAKE_CXX_FLAGS_DEBUG: -g -O0 -ggdb -fsanitize=address -fno-omit-frame-pointer -fno-optimize-sibling-calls
-- BusTub/test found valgrind at /usr/bin/valgrind
-- Configuring done
-- Generating done
-- Build files have been written to: /autograder/submission/project-handout/build

Scanning dependencies of target thirdparty_murmur3
Scanning dependencies of target gtest
[  2%] Building CXX object src/CMakeFiles/thirdparty_murmur3.dir/__/third_party/murmur3/MurmurHash3.cpp.o
[  4%] Building CXX object googletest-build/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
[  7%] Linking CXX shared library ../lib/libthirdparty_murmur3.so
[  7%] Built target thirdparty_murmur3
Scanning dependencies of target bustub_shared
[  9%] Building CXX object src/CMakeFiles/bustub_shared.dir/buffer/buffer_pool_manager.cpp.o
[ 11%] Building CXX object src/CMakeFiles/bustub_shared.dir/buffer/clock_replacer.cpp.o
[ 14%] Building CXX object src/CMakeFiles/bustub_shared.dir/catalog/column.cpp.o
[ 16%] Building CXX object src/CMakeFiles/bustub_shared.dir/catalog/schema.cpp.o
[ 19%] Building CXX object src/CMakeFiles/bustub_shared.dir/common/config.cpp.o
[ 21%] Building CXX object src/CMakeFiles/bustub_shared.dir/common/util/string_util.cpp.o
[ 23%] Building CXX object src/CMakeFiles/bustub_shared.dir/concurrency/lock_manager.cpp.o
[ 26%] Building CXX object src/CMakeFiles/bustub_shared.dir/concurrency/transaction_manager.cpp.o
[ 28%] Linking CXX static library ../../lib/libgtest.a
[ 28%] Built target gtest
[ 30%] Building CXX object src/CMakeFiles/bustub_shared.dir/container/hash/linear_probe_hash_table.cpp.o
Scanning dependencies of target gmock
[ 33%] Building CXX object googletest-build/googlemock/CMakeFiles/gmock.dir/src/gmock-all.cc.o
[ 35%] Building CXX object src/CMakeFiles/bustub_shared.dir/recovery/checkpoint_manager.cpp.o
[ 38%] Building CXX object src/CMakeFiles/bustub_shared.dir/recovery/log_manager.cpp.o
[ 40%] Building CXX object src/CMakeFiles/bustub_shared.dir/recovery/log_recovery.cpp.o
[ 42%] Linking CXX static library ../../lib/libgmock.a
[ 42%] Built target gmock
Scanning dependencies of target gtest_main
[ 45%] Building CXX object googletest-build/googletest/CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
[ 47%] Linking CXX static library ../../lib/libgtest_main.a
[ 50%] Building CXX object src/CMakeFiles/bustub_shared.dir/storage/disk/disk_manager.cpp.o
[ 50%] Built target gtest_main
[ 52%] Building CXX object src/CMakeFiles/bustub_shared.dir/storage/page/hash_table_block_page.cpp.o
[ 54%] Building CXX object src/CMakeFiles/bustub_shared.dir/storage/page/hash_table_header_page.cpp.o
Scanning dependencies of target gmock_main
[ 57%] Building CXX object googletest-build/googlemock/CMakeFiles/gmock_main.dir/src/gmock_main.cc.o
[ 59%] Building CXX object src/CMakeFiles/bustub_shared.dir/storage/page/header_page.cpp.o
[ 61%] Building CXX object src/CMakeFiles/bustub_shared.dir/storage/page/table_page.cpp.o
[ 64%] Linking CXX static library ../../lib/libgmock_main.a
[ 64%] Built target gmock_main
[ 66%] Building CXX object src/CMakeFiles/bustub_shared.dir/storage/table/table_heap.cpp.o
[ 69%] Building CXX object src/CMakeFiles/bustub_shared.dir/storage/table/table_iterator.cpp.o
[ 71%] Building CXX object src/CMakeFiles/bustub_shared.dir/storage/table/tuple.cpp.o
[ 76%] Building CXX object src/CMakeFiles/bustub_shared.dir/type/bigint_type.cpp.o
[ 76%] Building CXX object src/CMakeFiles/bustub_shared.dir/type/boolean_type.cpp.o
[ 78%] Building CXX object src/CMakeFiles/bustub_shared.dir/type/decimal_type.cpp.o
[ 80%] Building CXX object src/CMakeFiles/bustub_shared.dir/type/integer_parent_type.cpp.o
[ 83%] Building CXX object src/CMakeFiles/bustub_shared.dir/type/integer_type.cpp.o
[ 85%] Building CXX object src/CMakeFiles/bustub_shared.dir/type/smallint_type.cpp.o
[ 88%] Building CXX object src/CMakeFiles/bustub_shared.dir/type/timestamp_type.cpp.o
[ 90%] Building CXX object src/CMakeFiles/bustub_shared.dir/type/tinyint_type.cpp.o
[ 92%] Building CXX object src/CMakeFiles/bustub_shared.dir/type/type.cpp.o
[ 95%] Building CXX object src/CMakeFiles/bustub_shared.dir/type/value.cpp.o
[ 97%] Building CXX object src/CMakeFiles/bustub_shared.dir/type/varlen_type.cpp.o
[100%] Linking CXX shared library ../lib/libbustub_shared.so
[100%] Built target bustub_shared

[  3%] Built target thirdparty_murmur3
[  6%] Built target gtest
[ 10%] Built target gmock
[ 13%] Built target gmock_main
[ 66%] Built target bustub_shared
Scanning dependencies of target type_test
[ 68%] Building CXX object test/CMakeFiles/type_test.dir/type/type_test.cpp.o
[ 70%] Linking CXX executable type_test
[ 70%] Built target type_test
Scanning dependencies of target clock_replacer_test
[ 71%] Building CXX object test/CMakeFiles/clock_replacer_test.dir/buffer/clock_replacer_test.cpp.o
[ 73%] Linking CXX executable clock_replacer_test
[ 73%] Built target clock_replacer_test
Scanning dependencies of target grading_clock_replacer_test
[ 75%] Building CXX object test/CMakeFiles/grading_clock_replacer_test.dir/buffer/grading_clock_replacer_test.cpp.o
[ 76%] Linking CXX executable grading_clock_replacer_test
[ 76%] Built target grading_clock_replacer_test
Scanning dependencies of target grading_buffer_pool_manager_test
[ 78%] Building CXX object test/CMakeFiles/grading_buffer_pool_manager_test.dir/buffer/grading_buffer_pool_manager_test.cpp.o
[ 80%] Linking CXX executable grading_buffer_pool_manager_test
[ 80%] Built target grading_buffer_pool_manager_test
Scanning dependencies of target buffer_pool_manager_test
[ 81%] Building CXX object test/CMakeFiles/buffer_pool_manager_test.dir/buffer/buffer_pool_manager_test.cpp.o
[ 83%] Linking CXX executable buffer_pool_manager_test
[ 83%] Built target buffer_pool_manager_test
Scanning dependencies of target lock_manager_test
[ 85%] Building CXX object test/CMakeFiles/lock_manager_test.dir/concurrency/lock_manager_test.cpp.o
[ 86%] Linking CXX executable lock_manager_test
[ 86%] Built target lock_manager_test
Scanning dependencies of target hash_table_test
[ 88%] Building CXX object test/CMakeFiles/hash_table_test.dir/container/hash_table_test.cpp.o
[ 90%] Linking CXX executable hash_table_test
[ 90%] Built target hash_table_test
Scanning dependencies of target rwmutex_test
[ 91%] Building CXX object test/CMakeFiles/rwmutex_test.dir/common/rwmutex_test.cpp.o
[ 93%] Linking CXX executable rwmutex_test
[ 93%] Built target rwmutex_test
Scanning dependencies of target header_page_test
[ 95%] Building CXX object test/CMakeFiles/header_page_test.dir/table/header_page_test.cpp.o
[ 96%] Linking CXX executable header_page_test
[ 96%] Built target header_page_test
Scanning dependencies of target tuple_test
[ 98%] Building CXX object test/CMakeFiles/tuple_test.dir/table/tuple_test.cpp.o
[100%] Linking CXX executable tuple_test
[100%] Built target tuple_test
Scanning dependencies of target build-tests
Test project /autograder/submission/project-handout/build/test
  Test  #1: buffer_pool_manager_test
  Test  #2: clock_replacer_test
  Test  #3: grading_buffer_pool_manager_test
  Test  #4: grading_clock_replacer_test
  Test  #5: rwmutex_test
  Test  #6: lock_manager_test
  Test  #7: hash_table_test
  Test  #8: header_page_test
  Test  #9: tuple_test
  Test #10: type_test

Total Tests: 10
[100%] Built target build-tests

Scanning dependencies of target check-lint
Built target check-lint

[  5%] Built target thirdparty_murmur3
[ 94%] Built target bustub_shared
[100%] Built target gtest
Scanning dependencies of target check-clang-tidy
Enabled checks:
    bugprone-argument-comment
    bugprone-assert-side-effect
    bugprone-bool-pointer-implicit-conversion
    bugprone-copy-constructor-init
    bugprone-dangling-handle
    bugprone-exception-escape
    bugprone-fold-init-type
    bugprone-forward-declaration-namespace
    bugprone-forwarding-reference-overload
    bugprone-inaccurate-erase
    bugprone-incorrect-roundings
    bugprone-integer-division
    bugprone-lambda-function-name
    bugprone-macro-parentheses
    bugprone-macro-repeated-side-effects
    bugprone-misplaced-operator-in-strlen-in-alloc
    bugprone-misplaced-widening-cast
    bugprone-move-forwarding-reference
    bugprone-multiple-statement-macro
    bugprone-narrowing-conversions
    bugprone-parent-virtual-call
    bugprone-sizeof-container
    bugprone-sizeof-expression
    bugprone-string-constructor
    bugprone-string-integer-assignment
    bugprone-string-literal-with-embedded-nul
    bugprone-suspicious-enum-usage
    bugprone-suspicious-memset-usage
    bugprone-suspicious-missing-comma
    bugprone-suspicious-semicolon
    bugprone-suspicious-string-compare
    bugprone-swapped-arguments
    bugprone-terminating-continue
    bugprone-throw-keyword-missing
    bugprone-undefined-memory-manipulation
    bugprone-undelegated-constructor
    bugprone-unused-raii
    bugprone-unused-return-value
    bugprone-use-after-move
    bugprone-virtual-near-miss
    clang-analyzer-apiModeling.StdCLibraryFunctions
    clang-analyzer-apiModeling.TrustNonnull
    clang-analyzer-apiModeling.google.GTest
    clang-analyzer-core.CallAndMessage
    clang-analyzer-core.DivideZero
    clang-analyzer-core.DynamicTypePropagation
    clang-analyzer-core.NonNullParamChecker
    clang-analyzer-core.NonnilStringConstants
    clang-analyzer-core.NullDereference
    clang-analyzer-core.StackAddressEscape
    clang-analyzer-core.UndefinedBinaryOperatorResult
    clang-analyzer-core.VLASize
    clang-analyzer-core.builtin.BuiltinFunctions
    clang-analyzer-core.builtin.NoReturnFunctions
    clang-analyzer-core.uninitialized.ArraySubscript
    clang-analyzer-core.uninitialized.Assign
    clang-analyzer-core.uninitialized.Branch
    clang-analyzer-core.uninitialized.CapturedBlockVariable
    clang-analyzer-core.uninitialized.UndefReturn
    clang-analyzer-cplusplus.InnerPointer
    clang-analyzer-cplusplus.Move
    clang-analyzer-cplusplus.SelfAssignment
    clang-analyzer-deadcode.DeadStores
    clang-analyzer-nullability.NullPassedToNonnull
    clang-analyzer-nullability.NullReturnedFromNonnull
    clang-analyzer-nullability.NullableDereferenced
    clang-analyzer-nullability.NullablePassedToNonnull
    clang-analyzer-nullability.NullableReturnedFromNonnull
    clang-analyzer-optin.cplusplus.VirtualCall
    clang-analyzer-optin.mpi.MPI-Checker
    clang-analyzer-optin.osx.cocoa.localizability.EmptyLocalizationContextChecker
    clang-analyzer-optin.osx.cocoa.localizability.NonLocalizedStringChecker
    clang-analyzer-optin.performance.GCDAntipattern
    clang-analyzer-optin.performance.Padding
    clang-analyzer-optin.portability.UnixAPI
    clang-analyzer-osx.API
    clang-analyzer-osx.NumberObjectConversion
    clang-analyzer-osx.OSObjectRetainCount
    clang-analyzer-osx.ObjCProperty
    clang-analyzer-osx.SecKeychainAPI
    clang-analyzer-osx.cocoa.AtSync
    clang-analyzer-osx.cocoa.AutoreleaseWrite
    clang-analyzer-osx.cocoa.ClassRelease
    clang-analyzer-osx.cocoa.Dealloc
    clang-analyzer-osx.cocoa.IncompatibleMethodTypes
    clang-analyzer-osx.cocoa.Loops
    clang-analyzer-osx.cocoa.MissingSuperCall
    clang-analyzer-osx.cocoa.NSAutoreleasePool
    clang-analyzer-osx.cocoa.NSError
    clang-analyzer-osx.cocoa.NilArg
    clang-analyzer-osx.cocoa.NonNilReturnValue
    clang-analyzer-osx.cocoa.ObjCGenerics
    clang-analyzer-osx.cocoa.RetainCount
    clang-analyzer-osx.cocoa.RunLoopAutoreleaseLeak
    clang-analyzer-osx.cocoa.SelfInit
    clang-analyzer-osx.cocoa.SuperDealloc
    clang-analyzer-osx.cocoa.UnusedIvars
    clang-analyzer-osx.cocoa.VariadicMethodTypes
    clang-analyzer-osx.coreFoundation.CFError
    clang-analyzer-osx.coreFoundation.CFNumber
    clang-analyzer-osx.coreFoundation.CFRetainRelease
    clang-analyzer-osx.coreFoundation.containers.OutOfBounds
    clang-analyzer-osx.coreFoundation.containers.PointerSizedValues
    clang-analyzer-security.FloatLoopCounter
    clang-analyzer-security.insecureAPI.UncheckedReturn
    clang-analyzer-security.insecureAPI.bcmp
    clang-analyzer-security.insecureAPI.bcopy
    clang-analyzer-security.insecureAPI.bzero
    clang-analyzer-security.insecureAPI.getpw
    clang-analyzer-security.insecureAPI.gets
    clang-analyzer-security.insecureAPI.mkstemp
    clang-analyzer-security.insecureAPI.mktemp
    clang-analyzer-security.insecureAPI.rand
    clang-analyzer-security.insecureAPI.strcpy
    clang-analyzer-security.insecureAPI.vfork
    clang-analyzer-unix.API
    clang-analyzer-unix.Malloc
    clang-analyzer-unix.MallocSizeof
    clang-analyzer-unix.MismatchedDeallocator
    clang-analyzer-unix.Vfork
    clang-analyzer-unix.cstring.BadSizeArg
    clang-analyzer-unix.cstring.NullArg
    clang-analyzer-valist.CopyToSelf
    clang-analyzer-valist.Uninitialized
    clang-analyzer-valist.Unterminated
    google-build-explicit-make-pair
    google-build-namespaces
    google-build-using-namespace
    google-default-arguments
    google-explicit-constructor
    google-global-names-in-headers
    google-objc-avoid-throwing-exception
    google-objc-function-naming
    google-objc-global-variable-declaration
    google-readability-braces-around-statements
    google-readability-casting
    google-readability-function-size
    google-readability-namespace-comments
    google-readability-todo
    google-runtime-int
    google-runtime-operator
    google-runtime-references
    modernize-avoid-bind
    modernize-concat-nested-namespaces
    modernize-deprecated-headers
    modernize-deprecated-ios-base-aliases
    modernize-loop-convert
    modernize-make-shared
    modernize-make-unique
    modernize-pass-by-value
    modernize-raw-string-literal
    modernize-redundant-void-arg
    modernize-replace-auto-ptr
    modernize-replace-random-shuffle
    modernize-return-braced-init-list
    modernize-shrink-to-fit
    modernize-unary-static-assert
    modernize-use-auto
    modernize-use-bool-literals
    modernize-use-default-member-init
    modernize-use-emplace
    modernize-use-equals-default
    modernize-use-equals-delete
    modernize-use-noexcept
    modernize-use-nullptr
    modernize-use-override
    modernize-use-transparent-functors
    modernize-use-uncaught-exceptions
    modernize-use-using
    performance-faster-string-find
    performance-for-range-copy
    performance-implicit-conversion-in-loop
    performance-inefficient-algorithm
    performance-inefficient-string-concatenation
    performance-inefficient-vector-operation
    performance-move-const-arg
    performance-move-constructor-init
    performance-noexcept-move-constructor
    performance-type-promotion-in-math-fn
    performance-unnecessary-copy-initialization
    performance-unnecessary-value-param
    portability-simd-intrinsics
    readability-avoid-const-params-in-decls
    readability-braces-around-statements
    readability-const-return-type
    readability-container-size-empty
    readability-delete-null-pointer
    readability-deleted-default
    readability-else-after-return
    readability-function-size
    readability-identifier-naming
    readability-implicit-bool-conversion
    readability-inconsistent-declaration-parameter-name
    readability-isolate-declaration
    readability-misleading-indentation
    readability-misplaced-array-index
    readability-named-parameter
    readability-non-const-parameter
    readability-redundant-control-flow
    readability-redundant-declaration
    readability-redundant-function-ptr-dereference
    readability-redundant-member-init
    readability-redundant-preprocessor
    readability-redundant-smartptr-get
    readability-redundant-string-cstr
    readability-redundant-string-init
    readability-simplify-boolean-expr
    readability-simplify-subscript-expr
    readability-static-accessed-through-instance
    readability-static-definition-in-anonymous-namespace
    readability-string-compare
    readability-uniqueptr-delete-release
    readability-uppercase-literal-suffix


 Checking: /autograder/submission/project-handout/build/googletest-src/googlemock/src/gmock_main.cc
 Checking: /autograder/submission/project-handout/build/googletest-src/googlemock/src/gmock-all.cc
 Checking: /autograder/submission/project-handout/build/googletest-src/googletest/src/gtest_main.cc
 Checking: /autograder/submission/project-handout/build/googletest-src/googletest/src/gtest-all.cc
 Checking: /autograder/submission/project-handout/src/buffer/buffer_pool_manager.cpp
 Checking: /autograder/submission/project-handout/src/buffer/clock_replacer.cpp
 Checking: /autograder/submission/project-handout/src/catalog/column.cpp
 Checking: /autograder/submission/project-handout/src/catalog/schema.cpp
 Checking: /autograder/submission/project-handout/src/common/config.cpp
 Checking: /autograder/submission/project-handout/src/common/util/string_util.cpp
 Checking: /autograder/submission/project-handout/src/concurrency/lock_manager.cpp
 Checking: /autograder/submission/project-handout/src/concurrency/transaction_manager.cpp
 Checking: /autograder/submission/project-handout/src/container/hash/linear_probe_hash_table.cpp
 Checking: /autograder/submission/project-handout/src/recovery/checkpoint_manager.cpp
 Checking: /autograder/submission/project-handout/src/recovery/log_manager.cpp
 Checking: /autograder/submission/project-handout/src/recovery/log_recovery.cpp
 Checking: /autograder/submission/project-handout/src/storage/disk/disk_manager.cpp
 Checking: /autograder/submission/project-handout/src/storage/page/hash_table_block_page.cpp
 Checking: /autograder/submission/project-handout/src/storage/page/hash_table_header_page.cpp
 Checking: /autograder/submission/project-handout/src/storage/page/header_page.cpp
 Checking: /autograder/submission/project-handout/src/storage/page/table_page.cpp
 Checking: /autograder/submission/project-handout/src/storage/table/table_heap.cpp
 Checking: /autograder/submission/project-handout/src/storage/table/table_iterator.cpp
 Checking: /autograder/submission/project-handout/src/storage/table/tuple.cpp
 Checking: /autograder/submission/project-handout/src/type/bigint_type.cpp
 Checking: /autograder/submission/project-handout/src/type/boolean_type.cpp
 Checking: /autograder/submission/project-handout/src/type/decimal_type.cpp
 Checking: /autograder/submission/project-handout/src/type/integer_parent_type.cpp
 Checking: /autograder/submission/project-handout/src/type/integer_type.cpp
 Checking: /autograder/submission/project-handout/src/type/smallint_type.cpp
 Checking: /autograder/submission/project-handout/src/type/timestamp_type.cpp
 Checking: /autograder/submission/project-handout/src/type/tinyint_type.cpp
 Checking: /autograder/submission/project-handout/src/type/type.cpp
 Checking: /autograder/submission/project-handout/src/type/value.cpp
 Checking: /autograder/submission/project-handout/src/type/varlen_type.cpp
 Checking: /autograder/submission/project-handout/third_party/murmur3/MurmurHash3.cpp
 Checking: /autograder/submission/project-handout/test/buffer/clock_replacer_test.cpp
 Checking: /autograder/submission/project-handout/test/buffer/grading_clock_replacer_test.cpp
 Checking: /autograder/submission/project-handout/test/buffer/grading_buffer_pool_manager_test.cpp
 Checking: /autograder/submission/project-handout/test/buffer/buffer_pool_manager_test.cpp
 Checking: /autograder/submission/project-handout/test/concurrency/lock_manager_test.cpp
 Checking: /autograder/submission/project-handout/test/container/hash_table_test.cpp
 Checking: /autograder/submission/project-handout/test/common/rwmutex_test.cpp
 Checking: /autograder/submission/project-handout/test/table/header_page_test.cpp
 Checking: /autograder/submission/project-handout/test/table/tuple_test.cpp
 Checking: /autograder/submission/project-handout/test/type/type_test.cpp
[100%] Built target check-clang-tidy

Test Failed: build errored

Submission timed out in Gradescope

Hi @apavlo ! I got this after submitting the buffer pool manager. Your submission timed out. It took longer than 600 seconds to run.

Previously I was able to get the error message from the autograder that check-clang-tidy failed. Now it just timed out after I fixed these errors. Could you take a look?

Thank you so much for publishing the videos and projects. I really appreciate it!

How can I update the metadata of a Page

There are not public function that I can use to update the new fetched page's metadata in my buffer pool, is there any ways?

using page_slot = frame_id_t in BufferPoolManager page table and free list could make it clearer

It's unclear how frame_id_t space of values relates to the BufferPoolManager.

Using the term using page_slot = frame_id_t could simplify the understanding of what is happening with the free pages list and the pool itself.

Having page_slot = frame_id_t allows not having separate mapping.

More unit tests

Currently, there are not enough sample tests for lots of functions (eg: there is no unit test for BufferPoolManager::DeletePage operator). Some concurrency tests would also be helpful for the students.

I would be able to share some of the tests which I am currently using in my fork.

Confusion on loading hash table back from disk after restarting the DBMS in project 2

In the handout for project 2 task 1, it says

If you create a hash table, write its pages to disk, and then restart the DBMS, you should be able to load back the hash table from disk after restarting.

As far as I am concerned, the implementation should be something like this:

Before the DBMS shuts down, the hash table header page id should be stored somewhere on the disk as "metadata" for a table.
After restarting the DBMS, the system creates a hash table, passes the old header page id to the hash table instance and then gets all hash indices restored.

It seems that the LinearProbeHashTable class currently being given cannot get "header page id" to restore hash table. Is it my responsibility to implement this?

And the further problem is how to implement the storing and restoring old hash table header page id.

bustub/test/primer/starter_test.cpp: namespace bustub should be added

Improve exception safety with smart pointers

Would you like to wrap any pointer data members with the class template “std::unique_ptr”?

Update candidates:

can open more test case in gradescope?

Hi, professor. Can open more test case in gradescope for debug easier? Thanks! @apavlo

issue in cloning to private repo : readme updates

While trying to follow cloning steps in readme, i was getting following errors:

Counting objects: 210, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (164/164), done.
Writing objects: 100% (210/210), 250.81 KiB | 0 bytes/s, done.
Total 210 (delta 29), reused 210 (delta 29)
remote: Resolving deltas: 100% (29/29), done.
To [email protected]:ashishnegi/private-bustub.git
 ! [remote rejected] master -> master (shallow update not allowed)
 ! [remote rejected] origin/HEAD -> origin/HEAD (shallow update not allowed)
 ! [remote rejected] origin/master -> origin/master (shallow update not allowed)
error: failed to push some refs to '[email protected]:ashishnegi/private-bustub.git'

I ran following command to fix it:

git fetch --unshallow origin

Incorrect comment description for SimpleHashJoinTest

The SQL query describing the test for SimpleHashJoin Executor is from the previous test. Not a huge bug, but confused me for a bit.

bustub/test/execution/executor_test.cpp

Lines 257 to 258 in f581f10

 TEST_F(ExecutorTest, DISABLED_SimpleHashJoinTest) { 

 // INSERT INTO empty_table2 SELECT colA, colB FROM test_1 WHERE colA < 500

Convert #define to constexpr

The BusTub has a bunch of #define'd stuff. For example, there is config.h and more generally you can just search for define. We would prefer these to be constexpr instead.

To quote Matt, that gives us "proper namespace-scoping, type-checking, clang-tidy naming checks, and nicer behavior with IDEs".

Switch BusTub from TravisCI to CircleCI

TravisCI gives us around 5 concurrent builds across all our cmu-db repos. Unfortunately, in practice this means that BusTub and Terrier will fight each other when trying to build. It is annoying enough waiting for other PRs in the same repo to build; we'd rather not wait for PRs in a completely different repo to build too.

We can alleviate this by switching BusTub away from Travis.

If someone wants to write a .circleci/config.yml or something similar, go for it.

support std::unique_lock m(ReaderWriterLatch->Writer()) for page

read-write paths of RW latch needs the support of RAII

Why TableIterator Begin(Transaction *txn) only checks the first_page_id_ in class TableHeap?

In the implementation, it seems that it only checks the tuples in the first_page_id_. If it can not find any valid one, it would return EOF.

What if the TableHeap has multiple pages, all the tuples in the first page are deleted over a period of time but valid tuples do exist in the second page?

Is the GradeScope for non-CMU students is available yet?

This is a great course on databases. Thanks for making all these course materials available in public.

As the project 0 and 1 have passed the due date, I really hope I can test my solution against additional test cases to learn more about these projects.

Question to the design of disk manager

I would like to ask a simple question: Is disk manager in bustub thread safe?

I would try to give my answer: it is not thread safe.

reason 1: it has some int attributes.
reason 2: std::fstream is not thread safe, refering to C++ Standard

27.1.3 Thread safety [iostreams.thread-safety]

Concurrent access to a stream object [string.streams, file.streams], stream buffer object [stream.buffers], or C Library stream [c.files] by multiple threads may result in a data race [intro.multithread] unless otherwise specified [iostream.objects]. [Note: Data races result in undefined behavior [intro.multithread].

Following words are under the assumption, that disk manager is not thread safe. If the assumption is not the truth, please ignore the following words.

Then I would like to ask: why to design the disk manager not to be thread safe?

Yes. We can have a mutex outside of the disk manager doing the protection. In the Project 1, we should have a mutex to protect the disk manager inside of buffer manager.

There is a way maybe better than that: to use POXIS pread, pwrite to implement the ReadPage and WritePage functions. Since the page_id represents a unique part of a file and different page_ids share no overlap in file. POSIX pread, pwrite would be thread safe and easy to
be implemented. (Possible disadvantage: not modern C++.)

I am hoping and being grateful to see an answer. 😄

The motivation of the question:
I want to have low lock(mutex) granularity in my implementation: the buffer manager have to release it's global_mutex during ReadPage and WritePage these I/O operations. Then I came to this question and did some reading. I even have upload a version to gradescope and passed the concurrency test, in which the disk manager is not protected by any mutex.

Valgrind time-out in the grapescope test for lab1

I tried the original files and got the same result. Is there any tips I can solve that? Thanks!

anybody can share me with the project solution? for self study

just as a reference for self study, not for evil :)

Type subsystem

The type subsystem could probably be cleaned up significantly. Right now, there are a bunch of macros with hardcoded variable names inside them, and a lot of the logic between the different types (e.g. SmallintType vs BigintType) is essentially copy-and-pasted. You could probably replace essentially all the existing type files with a few well-written template functions.

Bug in debug routine `BPLUSTREE_TYPE::ToGraph`

BPLUSTREE_TYPE::ToGraph defined here https://github.com/cmu-db/bustub/blob/master/src/storage/index/b_plus_tree.cpp#L282

// Print leaves
for (int i = 0; i < inner->GetSize(); i++) {
  auto child_page = reinterpret_cast<BPlusTreePage *>(bpm->FetchPage(inner->ValueAt(i))->GetData());
  ToGraph(child_page, bpm, out);
  if (i > 0) {
    auto sibling_page = reinterpret_cast<BPlusTreePage *>(bpm->FetchPage(inner->ValueAt(i - 1))->GetData());
    if (!sibling_page->IsLeafPage() && !child_page->IsLeafPage()) {
      out << "{rank=same " << internal_prefix << sibling_page->GetPageId() << " " << internal_prefix
          << child_page->GetPageId() << "};\n";
    }
  }
}

We can see from this segment of codes that sibling_page is not UNPINED after FetchPage call. This may cause some bugs.

Git clone --depth leads to unsuccess git push using url

The README tells that clone this repo using --depth, however, after doing this, it will not be able to push into new repo using --mirror.

https://stackoverflow.com/questions/28983842/remote-rejected-shallow-update-not-allowed-after-changing-git-remote-url

Maybe it's a specific problem on url push, but it happens on my computer that the git push receives messages like "shallow update not allowed", just like the so link. I have solved it but it can be more helpful for other students if fixed.

Do we need both GetTablePageId and GetPageId?

GetPageId() in class Page and GetTablePageId() in class TablePage return the same thing. GetTablePageId() seems to be a duplicated method. Can we delete it?

What is the complementary operation of "ApplyDelete" in Project 4?

The complementary operation of Insert is ApplyDelete, but what is complementary operation of ApplyDelete? Is it Insert?

Suppose the last operation of transaction is Insert (inserted into slot rid_a in the log record). Then it starts to rollback and transactionManager::Abort function would append a ApplyDelete record (also with rid_a). Assume the the system crashes and these Insert and ApplyDelete are the last logs written on disk.

For the redo phase in recovery: if we perform Insert and ApplyDelete for slot rid_a, then record rid_a would be removed at the end. However, in the undo phase, how should we deal with the operation ApplyDelete? This operation is "CLR" type and we should not undo it. But it seems the system does not support this type of log record. If we decide undo this operation of ApplyDelete by Insert the tuple, the problem is Insert does not guarantee insert it at exact slot position rid_a (let us say it is inserted into rid_b now) so we might not undo the Insert with rid_a for the next step. How should we handle the ApplyDelete operation?

I also have another question about where should the log "ABORT" being appended? Should it being appended before any rollback logs (in the slides) or after rollback logs (in textbook and the codes in TransactionManager::Abort)?

bustub/test/primer/starter_test.cpp line45 and line75 it's better to use std::move()

bustub/test/primer/starter_test.cpp line45 and line75 it's better to use std::move()
std::unique_ptr<RowMatrix> sum_ptr = RowMatrixOperations::AddMatrices(move(mat1_ptr), move(mat2_ptr));

	TEST_F(ExecutorTest, DISABLED_SimpleHashJoinTest) {
	// INSERT INTO empty_table2 SELECT colA, colB FROM test_1 WHERE colA < 500