[SPARK-44742][PYTHON][DOCS] Add Spark version drop down to the PySpark doc site #42428

panbingkun · 2023-08-10T10:40:04Z

What changes were proposed in this pull request?

The pr aims to add Spark version drop down to the PySpark doc site.

Why are the changes needed?

Currently, PySpark documentation does not have a version dropdown. While by default we want people to land on the latest version, it will be helpful and easier for people to find docs if we have this version dropdown.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manual testing.

cd python/docs
make html

…k doc site

panbingkun · 2023-08-10T10:42:27Z

After patch:
This is not a complete list. If this format is possible, I will add all version links.

gengliangwang · 2023-08-11T00:34:53Z

Add Spark version drop down to the PySpark doc site

This is more like a new page instead of "drop down"

gengliangwang · 2023-08-11T00:35:54Z

python/docs/source/index.rst

@@ -24,7 +24,7 @@ PySpark Overview
 **Date**: |today| **Version**: |release|

 **Useful links**:
-|binder|_ | `GitHub <https://github.com/apache/spark>`_ | `Issues <https://issues.apache.org/jira/projects/SPARK/issues>`_ | |examples|_ | `Community <https://spark.apache.org/community.html>`_
+|binder|_ | `GitHub <https://github.com/apache/spark>`_ | `Issues <https://issues.apache.org/jira/projects/SPARK/issues>`_ | |examples|_ | `Community <https://spark.apache.org/community.html>`_ | `Historical versions of documentation <history/index.html>`_


Is there another website showing such a link? I think the wording is a bit too long, given that it is not supposed to be frequently used.

Okay, let me try turning it into a dropdown combobox form.

HyukjinKwon · 2023-08-11T00:57:48Z

Can we have a dropdown box like https://pandas.pydata.org/docs/

https://koalas.readthedocs.io/en/latest/

You could refer to https://github.com/pandas-dev/pandas/tree/main/doc/source

panbingkun · 2023-08-11T01:27:09Z

Can we have a dropdown box like https://pandas.pydata.org/docs/

https://koalas.readthedocs.io/en/latest/

You could refer to https://github.com/pandas-dev/pandas/tree/main/doc/source

Thanks, let me do it.

panbingkun · 2023-08-12T01:42:30Z

@gengliangwang @HyukjinKwon
What do you think of the following style?

…k doc site

panbingkun · 2023-08-12T01:50:42Z

python/docs/source/_static/versions.json

+        "name": "0.7.0",
+        "version": "0.7.0"
+    }
+]


Why are there no links for versions below 0.7.0? I found that there are no corresponding pyspark documents for versions below 0.7.0 in the historical directory.

https://spark.apache.org/docs

I think that is because we don't have PySpark before 0.7 :)

https://spark.apache.org/releases/spark-release-0-7-0.html

allisonwang-db · 2023-08-14T21:34:02Z

@panbingkun The latest screenshot looks good! I am wondering if we should 1) remove the preview docs from the version dropdown and 2) make it by default point to the latest released docs?

zhengruifeng · 2023-08-15T01:47:06Z

the new doc looks pretty good!

panbingkun · 2023-08-15T03:06:49Z

@panbingkun The latest screenshot looks good! I am wondering if we should 1) remove the preview docs from the version dropdown and 2) make it by default point to the latest released docs?

1.Okay, done.
2.When we release a new version and generate pyspark docs, the 'release' value will be set to RELEASE_VERSION,

spark/python/docs/source/conf.py

Lines 121 to 128 in 46580ab

    
           # The version info for the project you're documenting, acts as replacement for 
        
           # |version| and |release|, also used in various other places throughout the 
        
           # built documents. 
        
           # 
        
           # The short X.Y version. 
        
           version = 'master' 
        
           # The full version, including alpha/beta/rc tags. 
        
           release = os.environ.get('RELEASE_VERSION', version)

and the default value for dropbox has also been set to 'release' as shown:

https://github.com/apache/spark/pull/42428/files#diff-0711e6775ff3eb4e0fb47fc40afecfe946211842573dae2c5eb872bbe53df45fR2-R5

In the test below when I set RELEASE_VERSION to 3.4.1, the page will look like this:

The style seems to be what we want, or do we need to set its default value to latest?

…k doc site

zhengruifeng · 2023-08-16T00:43:53Z

@gengliangwang @HyukjinKwon would you mind taking another look when you find some time?

zhengruifeng · 2023-08-16T06:44:37Z

hi @panbingkun if we see the built doc here,

the drop down is like following, is this expected?

panbingkun · 2023-08-16T07:12:39Z

hi @panbingkun if we see the built doc here,

the drop down is like following, is this expected?

@gengliangwang Due to access issues with Ajax's cross origin, we need to open it locally in this way for test:
Open - n - a/Applications/Google Chrome. app/Content/MacOS/Google Chrome -- args -- user data dir="/tmp/chrome_dev_test" -- disable web security
😄

HyukjinKwon · 2023-08-21T00:40:01Z

python/docs/source/_static/versions.json

@@ -0,0 +1,278 @@
+[


I wonder if we better remove EOL releases ... but no strong opinion WDYT @srowen and @dongjoon-hyun ?

Yes, let's just start with latest versions even, as a convenience to switch. People can always find older versions manually, but we don't need to make them extra discoverable

We have to remember to update this with new releases - I wonder, do we have another document to add that step to?

Also I think we could even just retain the latest versions in each release branch?

Yeah, I will add this logic to release-build. sh, which is what I want to do in the next PR,
Or should we continue to do it in this PR, which is more suitable? 😄

HyukjinKwon · 2023-08-21T00:41:24Z

python/docs/source/_templates/version-switcher.html

@@ -0,0 +1,60 @@
+<div id="version-button" class="dropdown">


Let's put the license header:

HyukjinKwon

Looks pretty good otherwise, cc @gengliangwang and @sarutak FYI

srowen · 2023-08-21T01:35:48Z

python/docs/source/_static/versions.json

@@ -0,0 +1,278 @@
+[


Yes, let's just start with latest versions even, as a convenience to switch. People can always find older versions manually, but we don't need to make them extra discoverable

…k doc site

HyukjinKwon · 2023-08-23T02:31:13Z

Alright, let's see how it gose.

Merged to master and branch-3.5.

…k doc site ### What changes were proposed in this pull request? The pr aims to add Spark version drop down to the PySpark doc site. ### Why are the changes needed? Currently, PySpark documentation does not have a version dropdown. While by default we want people to land on the latest version, it will be helpful and easier for people to find docs if we have this version dropdown. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual testing. ``` cd python/docs make html ``` Closes #42428 from panbingkun/SPARK-44742. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 49ba244) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

xuanyuanking · 2023-08-29T05:47:50Z

@HyukjinKwon @panbingkun I reverted this change in branch-3.5 because it caused a doc build failure. Please assist in fixing this on the master branch. The issue can be reproduced by running the release script using the docs step in dry run mode:

preparing documents... failed
/usr/local/lib/python3.8/dist-packages/nbsphinx.py:1422: RuntimeWarning: You are using an unsupported version of pandoc (2.5).
Your version must be at least (2.14.2) but less than (4.0.0).
Refer to https://pandoc.org/installing.html.
Continuing with doubts...
  nbconvert.utils.pandoc.check_pandoc_version()

Warning, treated as error:
unsupported theme option 'navbar_end' given
make: *** [Makefile:35: html] Error 2
/opt/spark-rm/output/spark/docs/_plugins/copy_api_dirs.rb:130:in `<top (required)>': Python doc generation failed (RuntimeError)
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/external.rb:60:in `require'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/external.rb:60:in `block in require_with_graceful_fail'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/external.rb:57:in `each'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/external.rb:57:in `require_with_graceful_fail'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/plugin_manager.rb:89:in `block in require_plugin_files'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/plugin_manager.rb:87:in `each'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/plugin_manager.rb:87:in `require_plugin_files'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/plugin_manager.rb:21:in `conscientious_require'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/site.rb:131:in `setup'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/site.rb:36:in `initialize'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/commands/build.rb:30:in `new'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/commands/build.rb:30:in `process'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/command.rb:91:in `block in process_with_graceful_fail'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/command.rb:91:in `each'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/command.rb:91:in `process_with_graceful_fail'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/lib/jekyll/commands/build.rb:18:in `block (2 levels) in init_with_program'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `block in execute'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `each'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `execute'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary/program.rb:44:in `go'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/mercenary-0.4.0/lib/mercenary.rb:21:in `program'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/gems/jekyll-4.2.1/exe/jekyll:15:in `<top (required)>'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/bin/jekyll:23:in `load'
        from /opt/spark-rm/output/spark/docs/.local_ruby_bundle/ruby/2.7.0/bin/jekyll:23:in `<main>'

HyukjinKwon · 2023-08-29T10:29:03Z

Yeah that's fine. Let's revert

panbingkun · 2023-08-30T01:48:19Z

Let me investigate the cause.
When I downgraded the pydata_sphinx_theme version locally from 0.8.0 to 0.4.1, this error occurred again

So I will submit a new PR and upgrade to the pydata_sphinx_theme version in Docker.

HyukjinKwon · 2023-08-30T03:38:33Z

Sure, thanks for investigating this.

…o 0.8.0 in `spark-rm` Dockerfile ### What changes were proposed in this pull request? The pr is followup #42428. ### Why are the changes needed? To fix issue: When our `pydata_sphinx_theme` version is `0.4.1`, there may be issues with not recognizing some configuration items. <img width="927" alt="image" src="https://github.com/apache/spark/assets/15246973/7ec54bb2-8c28-4863-8374-b3a5369873fc"> ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual testing. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #42730 from panbingkun/SPARK-44742_FOLLOWUP. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…o 0.8.0 in `spark-rm` Dockerfile ### What changes were proposed in this pull request? The pr is followup #42428. ### Why are the changes needed? To fix issue: When our `pydata_sphinx_theme` version is `0.4.1`, there may be issues with not recognizing some configuration items. <img width="927" alt="image" src="https://github.com/apache/spark/assets/15246973/7ec54bb2-8c28-4863-8374-b3a5369873fc"> ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual testing. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #42730 from panbingkun/SPARK-44742_FOLLOWUP. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 1712515) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

[SPARK-44742][PYTHON][DOCS] Add Spark version drop down to the PySpar…

1b5482c

…k doc site

github-actions bot added the PYTHON label Aug 10, 2023

gengliangwang reviewed Aug 11, 2023

View reviewed changes

panbingkun added 2 commits August 12, 2023 09:45

[SPARK-44742][PYTHON][DOCS] Add Spark version drop down to the PySpar…

4969140

…k doc site

Merge branch 'master' into SPARK-44742

b56bcdb

panbingkun commented Aug 12, 2023

View reviewed changes

panbingkun requested a review from gengliangwang August 12, 2023 01:52

Merge branch 'master' into SPARK-44742

93fed94

panbingkun added 2 commits August 15, 2023 11:07

[SPARK-44742][PYTHON][DOCS] Add Spark version drop down to the PySpar…

c2e4a22

…k doc site

Merge branch 'master' into SPARK-44742

016c7d4

HyukjinKwon reviewed Aug 21, 2023

View reviewed changes

HyukjinKwon approved these changes Aug 21, 2023

View reviewed changes

srowen requested changes Aug 21, 2023

View reviewed changes

panbingkun added 3 commits August 22, 2023 09:32

Merge branch 'master' into SPARK-44742

d6e1038

[SPARK-44742][PYTHON][DOCS] Add Spark version drop down to the PySpar…

9dc0420

…k doc site

[SPARK-44742][PYTHON][DOCS] Add Spark version drop down to the PySpar…

0e10753

…k doc site

panbingkun requested a review from srowen August 22, 2023 06:12

HyukjinKwon closed this in 49ba244 Aug 23, 2023

panbingkun mentioned this pull request Aug 30, 2023

[SPARK-44742][PYTHON][DOCS][FOLLOWUP] Upgrade pydata_sphinx_theme to 0.8.0 in spark-rm Dockerfile #42730

Closed

panbingkun mentioned this pull request Sep 12, 2023

[SPARK-45122][DOCS] Automate updating versions.json #42881

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-44742][PYTHON][DOCS] Add Spark version drop down to the PySpark doc site #42428

[SPARK-44742][PYTHON][DOCS] Add Spark version drop down to the PySpark doc site #42428

panbingkun commented Aug 10, 2023

panbingkun commented Aug 10, 2023

gengliangwang commented Aug 11, 2023

gengliangwang Aug 11, 2023

panbingkun Aug 11, 2023

HyukjinKwon commented Aug 11, 2023 •

edited

panbingkun commented Aug 11, 2023 •

edited

panbingkun commented Aug 12, 2023

panbingkun Aug 12, 2023

zhengruifeng Aug 15, 2023

allisonwang-db commented Aug 14, 2023

zhengruifeng commented Aug 15, 2023

panbingkun commented Aug 15, 2023 •

edited

zhengruifeng commented Aug 16, 2023

zhengruifeng commented Aug 16, 2023

panbingkun commented Aug 16, 2023 •

edited

HyukjinKwon Aug 21, 2023

srowen Aug 21, 2023

panbingkun Aug 22, 2023

srowen Aug 22, 2023

panbingkun Aug 22, 2023 •

edited

HyukjinKwon Aug 21, 2023

HyukjinKwon left a comment

srowen Aug 21, 2023

HyukjinKwon commented Aug 23, 2023

xuanyuanking commented Aug 29, 2023 •

edited

HyukjinKwon commented Aug 29, 2023

panbingkun commented Aug 30, 2023 •

edited

HyukjinKwon commented Aug 30, 2023

[SPARK-44742][PYTHON][DOCS] Add Spark version drop down to the PySpark doc site #42428

[SPARK-44742][PYTHON][DOCS] Add Spark version drop down to the PySpark doc site #42428

Conversation

panbingkun commented Aug 10, 2023

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

panbingkun commented Aug 10, 2023

gengliangwang commented Aug 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HyukjinKwon commented Aug 11, 2023 • edited

panbingkun commented Aug 11, 2023 • edited

panbingkun commented Aug 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

allisonwang-db commented Aug 14, 2023

zhengruifeng commented Aug 15, 2023

panbingkun commented Aug 15, 2023 • edited

zhengruifeng commented Aug 16, 2023

zhengruifeng commented Aug 16, 2023

panbingkun commented Aug 16, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

panbingkun Aug 22, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HyukjinKwon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HyukjinKwon commented Aug 23, 2023

xuanyuanking commented Aug 29, 2023 • edited

HyukjinKwon commented Aug 29, 2023

panbingkun commented Aug 30, 2023 • edited

HyukjinKwon commented Aug 30, 2023

HyukjinKwon commented Aug 11, 2023 •

edited

panbingkun commented Aug 11, 2023 •

edited

panbingkun commented Aug 15, 2023 •

edited

panbingkun commented Aug 16, 2023 •

edited

panbingkun Aug 22, 2023 •

edited

xuanyuanking commented Aug 29, 2023 •

edited

panbingkun commented Aug 30, 2023 •

edited