Discussion about clickhouse support in the OAP about skywalking HOT 14 CLOSED

There is no official team to do this. This is very clear for years.
We have decided to move on our own database, BanyanDB.

If you want to do something, go ahead, and contribute to help others.

from skywalking.

We never rejected a pull request about it. No one takes the maintenance responsibility. No pull request happens.

from skywalking.

We have initially implemented some ideas. If you want to have further communication, you can send me an email.

from skywalking.

@sglztc Your comments have been deleted, all discussions on GitHub need to be in English. Slack CN channel is for Chinese.

from skywalking.

We have initially implemented some ideas. If you want to have further communication, you can send me an email.

Already contacted, the commercial version requires a fee

from skywalking.

What is the specific reason why clickhouse storage is not supported? Are there any pitfalls in this area so that I can avoid it when designing? Very much looking forward to the author's answer.

from skywalking.

We have initially implemented some ideas. If you want to have further communication, you can send me an email.

Can you share the table design of this piece? We can implement the code ourselves.

from skywalking.

What is the specific reason why clickhouse storage is not supported? Are there any pitfalls in this area so that I can avoid it when designing? Very much looking forward to the author's answer.

I think I have said this very clearly. There isn't a thing about why. No one is actually contributing and maintaining this in the upstream. Do you know who wants to do this? You said we rejected something, but we didn't.

from skywalking.

What is the specific reason why clickhouse storage is not supported? Are there any pitfalls in this area so that I can avoid it when designing? Very much looking forward to the author's answer.

I think I have said this very clearly. There isn't a thing about why. No one is actually contributing and maintaining this in the upstream. Do you know who wants to do this? You said we rejected something, but we didn't.

Hello, teacher wu-sheng, you have misunderstood me. What I want to know is are there any pitfalls in the process of adapting to clickhouse? I would like to know some specific reasons why clickhouse is not supported, because we are currently doing research and implementation in this area to avoid taking some detours. Do I only need to implement storage-clickhouse-plugin to use clickhouse storage? Do you still need to do a deeper transformation?

from skywalking.

What I want to know is are there any pitfalls in the process of adapting to clickhouse?

I don't know that.

I would like to know some specific reasons why clickhouse is not supported, because we are currently doing research and implementation in this area to avoid taking some detours

The only reason is what I mentioned. No one shows up to say they will implement it and maintain the implementation to make sure it is product ready.

Do I only need to implement storage-clickhouse-plugin to use clickhouse storage? Do you still need to do a deeper transformation?

I have no idea about the clickhouse. From the design perspective, SkyWalking exposes the module APIs for making implementation easier. So, yes, we hope you only need to implement that.
The reality is on the other side, sometimes, you will need annotations or some advanced flags on the kernel level to make the performance good. You can see Elasticsearch and JDBC relative flags are in the core module codes.

So, you can see, every time, clickhouse relative discussions fall into the same hole. People want me or someone in the SkyWalking maintenance team to tell, but the truth is we as a group neither know the clickhouse, nor use it.
So, there is no magic and short path. You have to understand both SkyWalking OAP kernel/details and the features of ClickHouse, and write design, coding and run performance tests to prove that this new storage option really is good.

from skywalking.

The existing offering is being kept private and commercial only, which also shows up, it would not be that easy.
So, you have to make your decision. I could only join the discussion if you are going to donate this to upstream and have at least two committers taking responsibility to maintain them, once it lacks maintenance, it will be removed like IoTDB and InfluxDB storage options. That is how open source community works.

from skywalking.

What I want to know is are there any pitfalls in the process of adapting to clickhouse?

I don't know that.

I would like to know some specific reasons why clickhouse is not supported, because we are currently doing research and implementation in this area to avoid taking some detours

The only reason is what I mentioned. No one shows up to say they will implement it and maintain the implementation to make sure it is product ready.

Do I only need to implement storage-clickhouse-plugin to use clickhouse storage? Do you still need to do a deeper transformation?

I have no idea about the clickhouse. From the design perspective, SkyWalking exposes the module APIs for making implementation easier. So, yes, we hope you only need to implement that. The reality is on the other side, sometimes, you will need annotations or some advanced flags on the kernel level to make the performance good. You can see Elasticsearch and JDBC relative flags are in the core module codes.

So, you can see, every time, clickhouse relative discussions fall into the same hole. People want me or someone in the SkyWalking maintenance team to tell, but the truth is we as a group neither know the clickhouse, nor use it. So, there is no magic and short path. You have to understand both SkyWalking OAP kernel/details and the features of ClickHouse, and write design, coding and run performance tests to prove that this new storage option really is good.

thanks

from skywalking.

By analyzing the storage source code of jdbc and elasticsearch, we can find that it is feasible to use clickhouse for storage. We can learn from elasticsearch storage and use clickhouse large-width table design to give full play to the multi-column advantages of clickhouse. It is still under continuous development. I hope it will give some inspiration to developers who are using clickhouse as storage.Personal suggestions are welcome to discuss together.

from skywalking.

Once you have a proposal about implementation details, please go for discussion to explain your design.
If you want further to make this in upstream(if accepted), you need to prepare SWIP.

from skywalking.