delta-io / website Goto Github PK

Currently https://delta.io/docs/ points to a landing page. The actual docs (https://docs.delta.io/latest/index.html) are available once you click on the "Docs" link in page https://delta.io/docs/.

I think https://delta.io/docs/ should point directly to the actual docs (https://docs.delta.io/latest/index.html)

We would like to add Teradata Vantage to the list of vendors who support Delta table using manifest on the site's integration page.
Teradata Vantage to Delta Lake Integration.docx

Complaint from a Delta user I ran into at a conference. Today you have to navigate to a specific file in the Delta github repository, in order to see the specification:
https://github.com/delta-io/delta/blob/master/PROTOCOL.md

This is not user-friendly, nor even super dev-friendly...

Fix the blog page redirects - for example a search on google goes to this non-existent page: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjXtP6H0oD4AhUSzIsKHZk-BTAQFnoECAkQAQ&url=https%3A%2F%2Fdelta.io%2Fblog-gallery%2Fpage%2F2%2F&usg=AOvVaw3hxmpTj3TGkl-_m5j2cwsP

The delta.io StackOverflow, Twitter, LinkedIn, and Slack icons do not click even though there are <div href="..." ...> references. This is possible due to the recent upgrades that we need to change how the top menu bar is generated. Note, the footer works fine.

It would appear that many (though not all) delta functions will work on a path based delta table (i.e. non hive) like:

delta.`/path/to/table`

It would be good to have this be a bit more clear in the documentation on the website regarding:

1. This pattern in general
2. What works
3. What does not work

Per https://docs.snowflake.com/en/user-guide/tables-external-intro.html#delta-lake-support, add Snowflake external tables Delta Lake support to Integrations

While opening https://delta.io/blog-gallery/boost-delta-lake-performance-with-data-skipping-and-z-order/ I ran into the following exception.

Remove the "docs|source" text from the blog cards.

On the https://delta.io/learn/videos/ page:

• Remove extraneous whitespace near the top
• Provide descriptions for each video series
• Move Delta Rust video series after Last week in a Byte
• Move Delta Lake DW Techniques to before Getting Started with Delta Lake

Descriptions

• Last Week in a Byte: Get the latest Delta Lake news and events ... a week late!
• Delta Lake Community Office Hours: Join our regular community office hours to discuss issues around the Delta Lake community
• Simon & Denny Ask Us Anything: Join Simon Whiteley (Advancing Analytics) and Denny Lee (Databricks) and ask your burning Delta Lake, Lakehouse questions
• Delta Lake Discussions with Denny Lee (D3L2): Watch the vidcast (or listen to this podcast) with Delta Lake customers and/or contributors
• Delta Lake Tech Talks: Join our regular technical webinar series on the latest Delta Lake features and integrations
• Delta Rust: Would you like to know more about the Delta Rust ecosystem? Watch these videos!
• Delta Lake Tutorials: Jumpstart your Delta Lake technical implementation with these tutorials
• Getting Started with Delta Lake: Jumpstart your Delta Lake concepts with this video series
• Delta Lake DW Techniques: Apply Data Warehousing techniques to your Delta Lake-based lakehouse.
• Under the Sediments: Dive deep into the internals and Delta Lake code-base
• Conference Talks: Watch these conference talks that are not on YouTube.

The social tiles (Groups, Slack, LinkedIn, Twitter) are missing the white tile background; can we please add them back in? cc @jakebellacera

We should separate the Integrations page to call out Delta Sharing with all of this integrations per https://www.linkedin.com/posts/willgirten_deltasharing-deltalake-opensource-activity-7009159907136446464-08Dm?utm_source=share&utm_medium=member_desktop

Hi @dennyglee,

As discussed in another issue post. We would like to have ASA added to the service list on page https://delta.io/integrations/.

Here is an example of ASA product page: https://learn.microsoft.com/en-us/azure/stream-analytics/write-to-delta-lake.

Thank you. Please let me know if you need more information.

Best,
Emma

Here's the link: https://www.cidrdb.org/cidr2023/papers/p92-jain.pdf

We should probably have a link to this whitepaper right on the homepage.

StreamNative Pulsar IO::Lakehouse provides both a sink and source for Delta Lake per their GitHub and blog.

We should include their integration into the Delta Lake Integrations page.

The roadmap on the website is currently outdated: https://delta.io/roadmap

We should either update it or remove it. Perhaps we can just make the roadmap page links to the individual project roadmaps.

This request is regarding this doc. https://docs.databricks.com/delta/presto-integration.html
"Do not use AWS Glue Crawler"

In February, Glue Delta Lake crawler was launched. Currently, the above warning is not applicable.
Is it possible for you to update this doc?

For details about Glue Delta Lake crawler, see https://aws.amazon.com/blogs/big-data/crawl-delta-lake-tables-using-aws-glue-crawlers/

@dennyglee

Update delta.io/roadmap to match issue 920

Currently, the Delta Lake contributing guide can be found at: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md. As Delta Lake has multiple repos, we need to create a single contributing guide that all repos can point to instead.

There is a section that needs to be updated on the Getting Help page under "Delta User Slack Channel".

The #deltalake-databricks, #delta-sharing and #deltalake-oss channels no longer exist on Slack. It is now:

• #deltalake-questions: Discuss topics about the Delta lake open source project and its development. https://www.delta.io
• #events: Share information about Delta Lake meetups and conference talks.
• #general: All things Delta Lake.
• #random: Non-work banter and water cooler conversation.

Native support for

• Glue 4.0: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-delta-lake.html
• EMR 6.9: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-690-release.html
• Athena SQL engine 3.0: https://docs.aws.amazon.com/athena/latest/ug/delta-lake-tables.html

The updates below all pertain to https://delta.io/community/!

Add Delta Lake's Spotify podcast under the "Join the Delta Lake Community" section: https://open.spotify.com/show/6YvPDkILtWfnJNTzJ9HsmW?si=282ba8186896469a

Add the following events to the "Check out the upcoming and most recent events section":

Delta Lake Community Office Hours (2022-10-20) - https://community.linuxfoundation.org/events/details/lfhq-delta-lake-presents-delta-lake-office-hours-october-20-2022/

Simon + Denny AUA: Data Mesh (2022-10-18) - https://community.linuxfoundation.org/events/details/lfhq-delta-lake-presents-simon-denny-ask-us-anything-data-mesh-october-18-2022/

Delta Lake Community Office Hours (2022-10-06) - https://community.linuxfoundation.org/events/details/lfhq-delta-lake-presents-delta-lake-office-hours-october-6-2022/

Delta Lake Community Office Hours (2022-09-22) - https://community.linuxfoundation.org/events/details/lfhq-delta-lake-presents-delta-lake-office-hours-september-22-2022/

Thank you very much! @dennyglee @MrPowers

Hi, we implemented the file system with ceph rgw for delta Lake through openstack swift API. The scheme of the path is ceph, The full path is ceph://<your cephrgw container>/<path to delta Table> . There is no problem using s3singledriverlogstore in logstore. We talked about this before in github.com/delta-io/delta #877 and #950, and we have sent this function to Maven centra repository. Here are the links to SBT and Maven.
SBT:
libraryDependencies += "io.github.nanhu-lab" % "hadoop-cephrgw" % "1.0.1"
Maven:

<dependency>

<groupId>io.github.nanhu-lab</groupId>

<artifactId>hadoop-cephrgw</artifactId>

<version>1.0.1</version>

</dependency>

I just went the the Delta Lake blog page and wanted to search for posts related to pandas. I went here (https://delta.io/blog) and then started scrolling through the pages looking for the relevant blog post. It's possible we should just put all the blogs on a single page. There aren't too many of them.

It'd also be nice to have a search bar that users can filter for given terms. It'd be cool to allow users to type in "pandas" and see all the related posts.

• AWS Glue
• AWS EMR

The delta.io website banner is linking to the Delta 2.1.0 release. This should be updated with the most recent release, Delta 2.1.1 - https://github.com/delta-io/delta/releases/tag/v2.1.1

@dennyglee @MrPowers

Here's an image that's being cut off on the preview section:

This image also gets cut off here: https://delta.io/blog

The delta.io homepage has the following "the latest" section:

Manually updating this section every time a blog post is published is a little tedious.

We should be able to grab the last 5 blog posts from listing the src/blog directory. Each blog post has metadata at the top of the index.mdx file. The thumbnail & title can be found from the metadata at the top of the file. Automating this will make it easier to publish blogs.

The number 1 tip to grow your LinkedIn follower count is to draw traffic from outside LinkedIn.

Suggestion #1: Make our names clickable and on the blog posts and have them go right to our follow pages.

Suggestion #2: Create a CTA at the end of the blog posts with a CTA to follow the Delta Lake LinkedIn page. Setup metrics to quantify success of this CTA.

LinkedIn is currently just grabbing the first image for the preview:

We would prefer if LinkedIn was showing the thumbnail, which is the following image:

The Delta Lake community page lacks the current set of Delta Lake committers.

Currently the key features is an image, this should be converted to json card list with additional information

Could you please re-organize the community dropdown menu to

• Remove "Delta Community", "Contributing", "Contributing to Documentation" and "Meetings" from the menu (for now) to clean up the dropdown
• Rename "Contributing Guidelines" to "Contributing"
• Rename "Answer Questions" to "Getting Help"
• Rename "Organization" to "Adoption"

cc @tdas

Currently the “releases” link points to the latest release of Delta Lake and it should point to the all releases (not the latest specific release).

We should add the Delta Sharing R-client to the delta.io integrations: https://github.com/zacdav-db/delta-sharing-r

The dropdown buttons don't click in an expected way, which was giving me the impression that there might be something wrong with the site. Think this should be an easy fix. Right now, the dropdown buttons only work if you click directly over the text. If you click on the button, but, not directly on the text, nothing happens. The following image shows where the button works and where the button does not work right now:

For integrations that only have docs or source code, the tags end up going the same line as opposed to be in separate lines per screenshot.

Let's fix it so the UI isn't so confusing and cluttered.

Per dependabot security issues, test and validate package updates

The Flink/Delta connector was released as part of Delta Connectors 0.4.0 per https://github.com/delta-io/connectors/releases/tag/v0.4.0. Please remove "(preview)".

https://github.com/badal-io/datastream-deltalake-connector

More info at: https://youtu.be/8rZDbqezs60 and https://databricks.com/blog/2022/02/03/google-datastream-integration-with-delta-lake-for-change-data-capture.html

Here's the current whitepapers section:

We should update this to link to both whitepapers:

• Analyzing and comparing Lakehouse storage systems
• Lakehouse architecture: Unifying data warehouses and advanced analytids

We currently do not have committers guidelines and requirements on the delta.io website, we need to add a page that provides that context.

We currently have a key features section on the homepage that highlights some of the nice Delta Lake features:

Perhaps we should create a delta.io/key-features pages with an exhaustive list of all the key features. It'd be nice if each key feature section linked to a blog post explaining the key feature in detail. The time travel key feature section could link to this blog post for example.

It'd be nice to have a page that clearly outlines all the reasons why the viewer should be seriously considering using Delta Lake.

Hello! Currently, the icons (Twitter, Slack, LinkedIn, etc) in the top right-hand corner of the website.

When there is a scheduled YouTube event, the community page cannot build and gives us the following error. We should get this fixed.

The current workaround is to flip the event to private (vs. public) temporarily during the build, process, publish, and then can flip the YouTube event back to public.

6:05:57 PM: success Rewriting compilation hashes - 0.000s
6:05:57 PM: error UNHANDLED REJECTION Couldn't find temp query result for "/community/".
6:05:57 PM: 
6:05:57 PM:   Error: Couldn't find temp query result for "/community/".
  
6:05:57 PM:   - page-data.ts:104 readPageQueryResult
6:05:57 PM:     [repo]/[gatsby]/src/utils/page-data.ts:104:11
6:05:57 PM:   
6:05:57 PM:   - runMicrotasks
6:05:57 PM:   
6:05:57 PM:   - task_queues:61 runNextTicks
6:05:57 PM:     node:internal/process/task_queues:61:5
6:05:57 PM:   
6:05:57 PM:   - timers:437 processImmediate
6:05:57 PM:     node:internal/timers:437:9
6:05:57 PM:   
6:05:57 PM:   - page-data.ts:121 writePageData
6:05:57 PM:     [repo]/[gatsby]/src/utils/page-data.ts:121:18
6:05:57 PM:   
6:05:57 PM:   - page-data.ts:228 
6:05:57 PM:     [repo]/[gatsby]/src/utils/page-data.ts:228:24
6:05:57 PM:   
6:05:57 PM: 
6:05:57 PM: not finished Writing page-data.json files to public directory - 0.027s
6:05:58 PM: ​
6:05:58 PM:   "build.command" failed                                        
6:05:58 PM: ────────────────────────────────────────────────────────────────
6:05:58 PM: ​
6:05:58 PM:   Error message
6:05:58 PM:   Command failed with exit code 1: npm run build (https://ntl.fyi/exit-code-1)

We need to update this section with the latest set of Delta Lake videos including Delta Lake YouTube, DAIS22, Cinco de Trino, Flink Forward 2022, Open Source Summit NA, ODSC Europe 2022, ODSC East 2022, and more.

Hi, it's a pleasure to communicate again. Last year we supported Ceph storage in delta (#11) , we ware grateful that you added that to delta.io/integrations. After that, we focused on the connection of Apache Beam and data lake. We noticed beam-deltalake, which only support users to read Delta Lake data from Beam. Because of dependency conflicts, it contains a lot of source code of Daltalake Standalone Reader.
Our beam-datalake support to connect Apache Beam and data lake, such as Delta Lake, Apache Hudi, Apache iceberg. With DataLakeIO, data from Apache Beam's pipelines can be written to data lake. It is also supported to read data from data lake into Apache Beam's pipeline.
We provide test cases for beam and delta connections in beam-delta-example , there are four main steps:

• 1.The beam-datalake project needs to be compiled and installed first.
• 2.Then, create a delta table using initDeltaTable(), which contains four fields: id, user_name, user_age, user_remark.
• 3.Then, use testDeltaWrite() to write the data to Delta Lake. In testDeltaWrite(), the simulated data is created, then the simulated data is converted by Apache Beam (converting the user_name to uppercase), and finally written to Delta Lake.
• 4.At last, use testDeltaRead() to read the data out of Delta Lake, and then filter according to the user_age, and write the data that meets the criteria to text.

beam-deltalake also mentioned our DataLakeIO in issues1, he said he wouldn't be continuing the project. So, We sincerely hope that our beam-datalake can be added to delta.io/integrations instead of his , and we will continue to optimize our features.

Thank you very much!