Git Product home page Git Product logo

ex_aliyun_ots's Introduction

ExAliyunOts

Coverage Status Module Version Hex Docs Total Download License Last Updated

Aliyun TableStore SDK for Elixir/Erlang

TableStore is a NoSQL database service built on Alibaba Cloud’s Apsara distributed operating system that can store and access large volumes of structured data in real time.

Installation

def deps do
  [
    {:ex_aliyun_ots, "~> 0.11"}
  ]
end

Configuration

Add these settings below into the config/ots.secret.exs file.

config :ex_aliyun_ots, :my_instance,
  name: "MyInstanceName",
  endpoint: "MyInstanceEndpoint",
  access_key_id: "MyAliyunRAMKeyID",
  access_key_secret: "MyAliyunRAMKeySecret",
  pool_size: 100,
  pool_count: 1

config :ex_aliyun_ots,
  instances: [:my_instance],
  debug: false,
  enable_tunnel: false
  • debug, optional, specifies whether to enable debug logger, by default it is false, and please DO NOT use debug mode in production.
  • enable_tunnel, optional, specifies whether to enable tunnel functions, there will startup tunnel related supervisor and registry when enable it, by default it is false.
  • pool_size, optional, number of connections to maintain in each pool, involved when use Finch as Tesla http adapter, see Finch module usage for details, defaults to 100.
  • pool_count, optional, number of pools to start, involved when use Finch as Tesla http adapter, see Finch module usage for details, defaults to 1.

Using ExAliyunOts

To use ExAliyunOts, a module that calls use ExAliyunOts has to be defined:

defmodule MyApp.TableStore do
  use ExAliyunOts, instance: :my_instance
end

This automatically defines some macros and functions in the MyApp.TableStore module, here are some examples:

import MyApp.TableStore

# Create table
create_table "table",
  [{"pk1", :integer}, {"pk2", :string}]

# Put row
put_row "table",
  [{"pk1", "id1"}],
  [{"attr1", 10}, {"attr2", "attr2_value"}],
  condition: condition(:expect_not_exist),
  return_type: :pk

# Search index
search "table", "index_name",
  search_query: [
  query: match_query("age", 28),
  sort: [
    field_sort("age", order: :desc)
  ]
]

# Local transaction
start_local_transaction "table", {"partition_key", "partition_value"}

ExAliyunOts API

There are two ways to use ExAliyunOts:

  • using macros and functions from your own ExAliyunOts module, like MyApp.TableStore.
  • using macros and functions from the ExAliyunOts module.

All defined functions and macros in ExAliyunOts are available and referable for your own ExAliyunOts module as well, except that the given arity of functions may different, because the instance parameter of each invoke request is NOT needed from your own ExAliyunOts module although the ExAliyunOts module defines it.

Ecto Adapter

Here is a Tablestore adapter for Ecto implementation as an alternative.

Supported Functions

  • Table Operations
  • Row Operations
  • Conditional update
  • Auto-increment function of the primary key column
  • Filter
  • Atomic counters
  • Search index
  • Local transaction
  • Tunnel service
  • Timeline mode

Supported API

  • Table

    • CreateTable
    • ListTable
    • DeleteTable
    • UpdateTable
    • DescribeTable
    • ComputeSplitPointsBySize
  • Row

    • PutRow
    • GetRow
    • UpdateRow
    • DeleteRow
    • GetRange
    • BatchGetRow
    • BatchWriteRow
  • SearchIndex

    • CreateSearchIndex
    • DescribeSearchIndex
    • DeleteSearchIndex
    • Search
    • ParallelScan
  • LocalTransaction

    • StartLocalTransaction
    • CommitTransaction
    • AbortTransaction
  • SDK built-in

    • IterateAllRange
    • IterateParallelScan
    • IterateSearch
    • Sequence
    • StreamParallelScan
    • StreamRange
    • StreamSearch
  • Tunnel service

  • Timeline mode

Thanks

Thanks very much for the help and support of the Alibaba TableStore offical support team.

References

Alibaba Tablestore product official references:

License

This project is licensed under the MIT license. Copyright (c) 2018- Xin Zou.

ex_aliyun_ots's People

Contributors

feng19 avatar hou8 avatar kianmeng avatar naupio avatar xinz avatar xosdy avatar yarnus avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

ex_aliyun_ots's Issues

Missing `ots.secret.exs` file

When running the mix deps.get command, got this error:

$ mix deps.get
** (File.Error) could not read file "/home/foobar/ex_aliyun_ots/config/ots.secret.exs": no such file or directory
    (elixir 1.11.1) lib/file.ex:354: File.read!/1
    (elixir 1.11.1) lib/config.ex:240: Config.__import__!/1
    (mix 1.11.1) lib/mix/config.ex:161: anonymous fn/2 in Mix.Config.__import__!/2
    (elixir 1.11.1) lib/enum.ex:2181: Enum."-reduce/3-lists^foldl/2-0-"/3
    (mix 1.11.1) lib/mix/config.ex:160: Mix.Config.__import__!/2
    (stdlib 3.13) erl_eval.erl:680: :erl_eval.do_apply/6

Fix readme some incorrect document

e.g.

 def test() do
    put_row "table_name",
      [{"key1", "id1"}],
      [{"name", "name1"}, {"age", 20}],
      condition: condition(RowExistence.expect_not_exist),
      return_type: :pk
end

condition(RowExistence.expect_not_exist) is incorrect sample.

Improve plainbuffer decoding

Here are some benchmark results for note.

Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.10.3
Erlang 22.3.4.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 42 s

Benchmarking deserialize row binary Commit_a96291c...
Benchmarking deserialize row binary new...
Benchmarking deserialize row binary old...

Name                                            ips        average  deviation         median         99th %
deserialize row binary new                 40393.59      0.0248 ms    ±43.77%      0.0220 ms      0.0680 ms
deserialize row binary Commit_a96291c        721.95        1.39 ms    ±12.75%        1.34 ms        2.10 ms
deserialize row binary old                   437.25        2.29 ms    ±11.38%        2.21 ms        3.25 ms

Comparison:
deserialize row binary new                 40393.59
deserialize row binary Commit_a96291c        721.95 - 55.95x slower +1.36 ms
deserialize row binary old                   437.25 - 92.38x slower +2.26 ms

Memory usage statistics:

Name                                     Memory usage
deserialize row binary new                    8.74 KB
deserialize row binary Commit_a96291c      1319.05 KB - 150.88x memory usage +1310.31 KB
deserialize row binary old                   10.58 KB - 1.21x memory usage +1.84 KB

**All measurements for memory usage were the same**
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.10.3
Erlang 22.3.4.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 42 s

Benchmarking deserialize rows binary Commit_a96291c...
Benchmarking deserialize rows binary new...
Benchmarking deserialize rows binary old...

Name                                             ips        average  deviation         median         99th %
deserialize rows binary new                 34876.97      0.0287 ms    ±38.19%      0.0260 ms      0.0750 ms
deserialize rows binary old                   320.30        3.12 ms    ±11.28%        3.06 ms        4.34 ms
deserialize rows binary Commit_a96291c        298.79        3.35 ms    ±10.86%        3.27 ms        4.55 ms

Comparison:
deserialize rows binary new                 34876.97
deserialize rows binary old                   320.30 - 108.89x slower +3.09 ms
deserialize rows binary Commit_a96291c        298.79 - 116.73x slower +3.32 ms

Memory usage statistics:

Name                                           average  deviation         median         99th %
deserialize rows binary new                   13.17 KB     ±0.00%       13.17 KB       13.17 KB
deserialize rows binary old                  222.38 KB     ±0.08%      222.45 KB      222.50 KB
deserialize rows binary Commit_a96291c      2884.14 KB     ±0.00%     2884.14 KB     2884.14 KB

Comparison:
deserialize rows binary new                   13.17 KB
deserialize rows binary old                  222.38 KB - 16.88x memory usage +209.21 KB
deserialize rows binary Commit_a96291c      2884.14 KB - 218.96x memory usage +2870.97 KB
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.10.3
Erlang 22.3.4.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 42 s

Benchmarking serialize put_row Commit_a96291c...
Benchmarking serialize put_row new...
Benchmarking serialize put_row old...

Name                                       ips        average  deviation         median         99th %
serialize put_row Commit_a96291c        449.93        2.22 ms     ±9.40%        2.16 ms        3.09 ms
serialize put_row new                   422.20        2.37 ms    ±12.79%        2.27 ms        3.60 ms
serialize put_row old                   346.91        2.88 ms    ±12.32%        2.77 ms        4.23 ms

Comparison:
serialize put_row Commit_a96291c        449.93
serialize put_row new                   422.20 - 1.07x slower +0.146 ms
serialize put_row old                   346.91 - 1.30x slower +0.66 ms

Memory usage statistics:

Name                                Memory usage
serialize put_row Commit_a96291c        83.62 KB
serialize put_row new                   73.54 KB - 0.88x memory usage -10.07813 KB
serialize put_row old                 1884.48 KB - 22.54x memory usage +1800.87 KB

**All measurements for memory usage were the same**
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.10.3
Erlang 22.3.4.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 42 s

Benchmarking serialize primary keys Commit_a96291c...
Benchmarking serialize primary keys new...
Benchmarking serialize primary keys old...

Name                                            ips        average  deviation         median         99th %
serialize primary keys new                  18.62 K       53.71 μs    ±28.89%       49.98 μs      127.98 μs
serialize primary keys Commit_a96291c       18.35 K       54.50 μs    ±28.88%       50.98 μs      130.98 μs
serialize primary keys old                  17.78 K       56.23 μs    ±26.37%       51.98 μs      130.60 μs

Comparison:
serialize primary keys new                  18.62 K
serialize primary keys Commit_a96291c       18.35 K - 1.01x slower +0.79 μs
serialize primary keys old                  17.78 K - 1.05x slower +2.52 μs

Memory usage statistics:

Name                                     Memory usage
serialize primary keys new                    1.63 KB
serialize primary keys Commit_a96291c         1.98 KB - 1.22x memory usage +0.35 KB
serialize primary keys old                    4.88 KB - 2.99x memory usage +3.25 KB

**All measurements for memory usage were the same**

Improve crc

After this enhancement, we find a way to improve further for crc_int32/crc_int64, use bitstring in pattern match to tail recursion and calculate the final int value, here is a benchmark result for note:

Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.11.4
Erlang 23.3

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 28 s

Benchmarking crc_int32 with :binary.at...
Benchmarking crc_int32(Commit 938fdd9) with :binary.at...

Name                                                ips        average  deviation         median         99th %
crc_int32 with :binary.at                        2.14 M      467.34 ns  ±3860.42%           0 ns        1000 ns
crc_int32(Commit 938fdd9) with :binary.at        1.96 M      511.15 ns  ±4501.31%           0 ns        1000 ns

Comparison:
crc_int32 with :binary.at                        2.14 M
crc_int32(Commit 938fdd9) with :binary.at        1.96 M - 1.09x slower +43.81 ns

Memory usage statistics:

Name                                         Memory usage
crc_int32 with :binary.at                            40 B
crc_int32(Commit 938fdd9) with :binary.at            56 B - 1.40x memory usage +16 B

**All measurements for memory usage were the same**
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.11.4
Erlang 23.3

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 28 s

Benchmarking crc_int64 with :binary.at...
Benchmarking crc_int64(Commit 938fdd9) with :binary.at...

Name                                                ips        average  deviation         median         99th %
crc_int64 with :binary.at                        1.42 M      704.10 ns  ±2728.87%        1000 ns        1000 ns
crc_int64(Commit 938fdd9) with :binary.at        1.25 M      802.02 ns  ±1612.12%        1000 ns        1000 ns

Comparison:
crc_int64 with :binary.at                        1.42 M
crc_int64(Commit 938fdd9) with :binary.at        1.25 M - 1.14x slower +97.92 ns

Memory usage statistics:

Name                                         Memory usage
crc_int64 with :binary.at                            40 B
crc_int64(Commit 938fdd9) with :binary.at            56 B - 1.40x memory usage +16 B

**All measurements for memory usage were the same**

Improve crc/plainbuffer calcuate

Use :binary.at/2 to instead of :lists.nth/2 to improve ExAliyunOts.CRC module, then make a better serialize performance for WRITE operations.

Here are some benchmark results for compare:

Run mix run benchmark/crc.exs:

Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.11.4
Erlang 23.3

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 24 s

Benchmarking crc_int8 with :binary.at...
Benchmarking crc_int8 with :lists.nth...

Name                               ips        average  deviation         median         99th %
crc_int8 with :binary.at        6.51 M       0.154 μs   ±436.79%           0 μs           1 μs
crc_int8 with :lists.nth        0.60 M        1.68 μs    ±97.41%           2 μs           2 μs

Comparison:
crc_int8 with :binary.at        6.51 M
crc_int8 with :lists.nth        0.60 M - 10.92x slower +1.52 μs
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.11.4
Erlang 23.3

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 24 s

Benchmarking crc_int32 with :binary.at...
Benchmarking crc_int32 with :lists.nth...

Name                                ips        average  deviation         median         99th %
crc_int32 with :binary.at        2.00 M        0.50 μs  ±4902.26%           0 μs           1 μs
crc_int32 with :lists.nth        0.23 M        4.38 μs   ±302.31%           4 μs          18 μs

Comparison:
crc_int32 with :binary.at        2.00 M
crc_int32 with :lists.nth        0.23 M - 8.74x slower +3.88 μs
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.11.4
Erlang 23.3

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 24 s

Benchmarking crc_int64 with :binary.at...
Benchmarking crc_int64 with :lists.nth...

Name                                ips        average  deviation         median         99th %
crc_int64 with :binary.at        1.24 M        0.81 μs  ±3116.51%           1 μs           1 μs
crc_int64 with :lists.nth      0.0845 M       11.84 μs    ±67.44%          11 μs          37 μs

Comparison:
crc_int64 with :binary.at        1.24 M
crc_int64 with :lists.nth      0.0845 M - 14.64x slower +11.03 μs
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.11.4
Erlang 23.3

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 24 s

Benchmarking crc_string with :binary.at...
Benchmarking crc_string with :lists.nth...

Name                                 ips        average  deviation         median         99th %
crc_string with :binary.at      176.64 K        5.66 μs   ±159.87%        4.97 μs       21.97 μs
crc_string with :lists.nth        3.45 K      289.77 μs    ±16.46%      272.97 μs      493.97 μs

Comparison:
crc_string with :binary.at      176.64 K
crc_string with :lists.nth        3.45 K - 51.18x slower +284.11 μs

Run mix run benchmark/plainbuffer.exs (ignore "deserialize" parts):

...
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.11.4
Erlang 23.3

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 28 s

Benchmarking serialize put_row Commit_3a4a442...
Benchmarking serialize put_row new...

Name                                       ips        average  deviation         median         99th %
serialize put_row new                   6.36 K       0.157 ms    ±17.40%       0.148 ms        0.28 ms
serialize put_row Commit_3a4a442        0.40 K        2.50 ms     ±7.11%        2.45 ms        3.14 ms

Comparison:
serialize put_row new                   6.36 K
serialize put_row Commit_3a4a442        0.40 K - 15.87x slower +2.34 ms

Memory usage statistics:

Name                                Memory usage
serialize put_row new                   75.23 KB
serialize put_row Commit_3a4a442        75.23 KB - 1.00x memory usage +0 KB

**All measurements for memory usage were the same**
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.11.4
Erlang 23.3

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 28 s

Benchmarking serialize primary keys Commit_3a4a442...
Benchmarking serialize primary keys new...

Name                                            ips        average  deviation         median         99th %
serialize primary keys new                 261.67 K        3.82 μs   ±749.33%           3 μs          18 μs
serialize primary keys Commit_3a4a442       17.23 K       58.03 μs    ±26.88%          54 μs         136 μs

Comparison:
serialize primary keys new                 261.67 K
serialize primary keys Commit_3a4a442       17.23 K - 15.19x slower +54.21 μs

Memory usage statistics:

Name                                     Memory usage
serialize primary keys new                    1.63 KB
serialize primary keys Commit_3a4a442         1.63 KB - 1.00x memory usage +0 KB

**All measurements for memory usage were the same**
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.11.4
Erlang 23.3

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 2 s
parallel: 1
inputs: none specified
Estimated total run time: 28 s

Benchmarking serialize for update row Commit_3a4a442...
Benchmarking serialize for update row new...

Name                                              ips        average  deviation         median         99th %
serialize for update row new                   600.99        1.66 ms     ±8.38%        1.62 ms        2.16 ms
serialize for update row Commit_3a4a442         38.68       25.86 ms     ±2.60%       25.75 ms       28.77 ms

Comparison:
serialize for update row new                   600.99
serialize for update row Commit_3a4a442         38.68 - 15.54x slower +24.19 ms

Memory usage statistics:

Name                                       Memory usage
serialize for update row new                  796.73 KB
serialize for update row Commit_3a4a442       796.80 KB - 1.00x memory usage +0.0703 KB

**All measurements for memory usage were the same**

SSL version with OTP 23

After upgrade with otp 23, since there are some SSL versions are removed in this version, we will occur the following error:

2020-05-15 10:50:34.481 [error] ** ExAliyunOts occur an error: %ExAliyunOts.Error{code: "OTSClientUnknownError", datetime: nil, http_status_code: 400, message: {:options, {:sslv3, {:versions, [:"tlsv1.2", :"tlsv1.1", :tlsv1, :sslv3]}}}, request_id: nil}

So far we force on using tlsv1.2 and tlsv1.3 to be compatible with the main recommend.

Error running benchmark

Benchmarking doesn't work. Do we need actual Aliyun TableStore instance?

$ mix run benchmark/crc.exs
.....
** (Mix) Could not start application ex_aliyun_ots: exited in: ExAliyunOts.Application.start(:normal, [])
    ** (EXIT) an exception was raised:
        ** (ExAliyunOts.RuntimeError) Error: "Not found any instances configuration"
            (ex_aliyun_ots 0.9.0) lib/ex_aliyun_ots/application.ex:26: ExAliyunOts.Application.child_spec/3
            (ex_aliyun_ots 0.9.0) lib/ex_aliyun_ots/application.ex:16: ExAliyunOts.Application.start/2
            (kernel 7.0) application_master.erl:277: :application_master.start_it_old/4

Enhance SearchIndex functions

New features:

  • Support virtual field (see product document)
  • Support top rows aggregation
  • Support percentiles histogram

Related changes for golang implement, see search.proto changes.

The portions of official product document are not updated yet.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.