SymPy 2022 Documentation Theme Survey Analysis

Methodology

SymPy ran a user survey about its documentation theme from February 5-19, 2022. The primary purpose of the survey was to guide the selection of a Sphinx theme for the SymPy Documentation at https://docs.sympy.org.

A total of 22 people responded. The survey was done on Google Surveys and was shared on the SymPy public mailing list, the @SymPy Twitter account, and a SymPy discussion on GitHub. The survey consisted of 14 questions, all of which were optional. The results of these responses are summarized here. We would like to thank everyone who took and shared the survey.

Four themes were chosen based on factors such as layout, navigation, performance, and accessibility for evaluation by the SymPy community. Each theme was "prototyped" by

• Applying the theme to the SymPy dev documentation
• Removing SymPy Live, which has several problems, is planned to be removed in the live documentation, and affects the formatting of the documentation site due to importing a style sheet
• Hosting them on GitHub Pages

No attempt was made to customize the four themes because that is anticipated to be a time-consuming process with both technical (styling) and consensus-building components. Respondents were thus encouraged to focus on the layout, navigation, and interactive features, rather than the exact styling, for example colors.

For each of the four themes, respondents were asked to

• Rate the theme's usefulness of a scale of 1 (Not very useful) to 4 (Very useful)
• Share what they liked and disliked

Summary

A detailed analysis of the responses is provided below. At a high level, there are three main takeaways from the results.

1. The themes can be divided into three ratings categories, where the rating scale was 1 (Not very useful) to 4 (Very useful):

   1. Highest: Furo at 2.95.
   2. Middle: PyData and Book, nearly tied at 2.85 and 2.86, respectively.
   3. Lowest: Read the Docs (RTD) at 2.47.

2. Most comments about themes, both likes and dislikes, were about formatting, look and feel, and navigation.

3. We should proceed with the Furo theme, customizing it to address respondents' dislikes about its formatting. We can keep the PyData and Book themes in mind as backup options.

import warnings
warnings.filterwarnings('ignore')

import pandas
import seaborn
import matplotlib.pyplot as plt

import textwrap
import pprint as pp

# Set the plot format to SVG for better quality plots
from matplotlib_inline.backend_inline import set_matplotlib_formats
set_matplotlib_formats('retina')

%matplotlib inline

df = pandas.read_csv('theme-responses.csv')

# Set up columns, themes, colors
timestamp, experience_level, pydata_rating, pydata_like, pydata_dislike, book_rating, book_like, book_dislike, furo_rating, furo_like, furo_dislike, rtd_rating, rtd_like, rtd_dislike, other_comments,  = df.columns

rating_cols = [pydata_rating, book_rating, furo_rating, rtd_rating]

themes = ["PyData", "Book", "Furo", "RTD"]
n_themes = len(themes)
theme_colors = ["blue", "red", "yellow", "green"]

n_responses = len(df)

Experience Level

The first question asked the respondents to place their SymPy experience level on a scale of 1 to 5, with 1 being "novice user" and 5 being "expert user".

Most respondents self-reported a mid-level experience with SymPy.

n_nr = df[experience_level].isna().sum()

experience_cat_nr = pandas.Series({'0': n_nr})
experience_cats_r = df[experience_level].dropna().astype(int).value_counts(sort=False).sort_index()
experience_cats = experience_cat_nr.append(experience_cats_r)

percents_experience_cats = ["%.1f%%" % i for i in experience_cats/n_responses*100]

ax = seaborn.countplot(df[experience_level].fillna(0).astype(int))
ax.bar_label(ax.containers[0], percents_experience_cats, label_type='center')
ax.set_xticklabels(['No response', "1 Novice user", "2", "3", "4", "5 Expert user"])

max_width = 10

# Split x tick labels across multiple lines
# https://stackoverflow.com/questions/57144682/split-string-xticks-into-multiple-lines-matplotlib
xtl = ax.set_xticklabels(textwrap.fill(x.get_text(), max_width) for x in ax.get_xticklabels())

Theme Ratings

The survey asked respondents to rate the usefulness of four themes on a 1-4 scale, with 1 being Not very useful and 4 being Very useful. The mean and standard deviation of the rating for each theme are expressed numerically and graphically as:

df[rating_cols].describe().transpose()[["mean","std"]].round(2)

	mean	std
How useful is the PyData theme?	2.85	0.75
How useful is the Book theme?	2.86	0.79
How useful is the Furo theme?	2.95	1.02
How useful is the Read the Docs (RTD) theme?	2.47	0.90

# So seaborn can automatically plot standard deviations as error bars,
# combine all ratings into one column, paired with theme
all_themes = []
for theme in themes:
    this_theme = [theme] * n_responses
    all_themes += this_theme

all_ratings = []
for col in rating_cols:
    this_theme_ratings = list(df[col])
    all_ratings += this_theme_ratings

df_combined = pandas.DataFrame(list(zip(all_themes, all_ratings)), columns = ['theme', 'rating'])

rating_min = 1
rating_max = 4
num_bins = rating_max - rating_min + 1
rating_values = range(rating_min, rating_max + 1)

t = seaborn.barplot(
    data=df_combined, 
    x="theme", 
    y="rating",
    capsize=0.2,
    errwidth=0.5,
    palette=theme_colors, 
    alpha=.6, 
)
t.set_yticks(rating_values)
t.bar_label(t.containers[0], label_type = 'center', fmt='%.2f')

t.set(xlabel='', ylabel='How useful is each theme?\n1= Not very useful             Very useful = 4')
t.set(ylim=(rating_min,rating_max))

t.grid(False)

Furo is the highest-rated theme by about 0.1 points. PyData and Book are virtually tied for second place. Read the Docs is rated lowest, about 0.5 points below Furo.

Rating Distribution for Themes

For each theme, a histogram displays the count of responses for each rating level, from 1 to 4, and the dashed vertical line indicates the mean rating.

## Functions to determine complimentary color
## https://stackoverflow.com/questions/40233986/python-is-there-a-function-or-formula-to-find-the-complementary-colour-of-a-rgb

# Sum of the min & max of (a, b, c)
def hilo(a, b, c):
    if c < b: b, c = c, b
    if b < a: a, b = b, a
    if c < b: b, c = c, b
    return a + c

# Get complimentary color
def complement(r, g, b):
    k = hilo(r, g, b)
    return tuple(k - u for u in (r, g, b))

import matplotlib
import matplotlib.ticker as mticker

fig, axes = plt.subplots(n_themes, figsize=(5, 15))
plt.subplots_adjust(hspace = 0)

for theme_num, theme in enumerate(themes):
    graph = seaborn.histplot(
    ax=axes[theme_num],
    data=df,
    x = rating_cols[theme_num],
    bins = num_bins,
    binrange=[rating_min,rating_max],
    color=theme_colors[theme_num],
    alpha = 0.6,
    edgecolor="white"
    )
    
    # Add vertical line at mean of each theme

    # Get RGB of bar's color
    bar_rgb = matplotlib.colors.to_rgb(theme_colors[theme_num])
    line_rgb = complement(*bar_rgb)

    #   https://datavizpyr.com/how-to-add-a-mean-median-line-to-a-seaborn-displot/
    graph.axvline(x=df[rating_cols[theme_num]].mean(),
        ls='--',
        color=line_rgb,
        ymax = 0.95
        )
    graph.set(ylabel=themes[theme_num] + " count")
    graph.grid(False)             # remove gridlines
    graph.set(yticklabels=[])     # remove y-axis tick labels

    # Add labels to bars: Percents
    theme_cats = df[rating_cols[theme_num]].dropna().astype(int).value_counts(sort=False).sort_index()

    # Ensure each bar has an entry in list
    denom = sum(theme_cats) / 100
    percents_cats = []
    for cat in range(rating_min, rating_max + 1):
        if cat in theme_cats:
            pct = theme_cats[cat] / denom
            percents_cats += ["%.0f%%" % pct]
        else:
            percents_cats += [""]

    l = graph.bar_label(
        graph.containers[0], 
        percents_cats, 
        label_type= 'center',
        # color=line_rgb,
        # color="black",
        bbox=dict(
            fc = "white", 
            lw = 1,
            )
        )

    # Hide tick marks by making them zero length
    graph.tick_params(length = 0)

    if theme_num in range(1, n_themes - 2):
        # For graphs in the middle (neither top nor bottom),
        #   remove x axis and tick labels
        graph.set(xlabel='')
        graph.set_xticklabels([])
    else:
        # For graphs at top and bottom,
        #   show x-axis title and tick labels

        # Fixing yticks with matplotlib.ticker "FixedLocator"
        #   https://stackoverflow.com/questions/63723514/userwarning-fixedformatter-should-only-be-used-together-with-fixedlocator
        label_format = '{:,.0f}'
        ticks_loc = graph.get_xticks().tolist()
        graph.xaxis.set_major_locator(mticker.FixedLocator(ticks_loc))
        graph.set_xticklabels([label_format.format(x) for x in ticks_loc])

        # Center labels on bars (columns)
        #   https://stackoverflow.com/questions/63516973/how-can-i-mark-xticks-at-the-center-of-bins-for-a-seaborn-distplot
        mids = [rect.get_x() + rect.get_width() / 2 for rect in graph.patches]
        graph.set_xticks(mids)

        graph.set(xlabel='How useful is each theme?')
        graph.set_xticklabels(['1\nNot very','2\n','3\n','4\nVery'])

        if theme_num == 0:
            graph.xaxis.set_ticks_position("top")
            graph.xaxis.set_label_position("top")

For Furo, the mode is 4, Very useful. The mode of the other three themes is 3.

Furo theme

Given that Furo is the highest-rated theme, it is worth considering other factors before deciding to proceed with it.

Comments about Furo

Here are consolidated lists of highlights from what respondents liked and disliked about Furo.

• Like
  • Clean look and clear font
  • Left and right sidebars for site and page navigation, respectively
  • Has both light and dark themes, and is easy to switch between them
  • Dislike
  • Colors are distracting (for example, behind code blocks), too dark without enough contrast
  • Bold and highlighting seem a little cartoonish
  • Not more widely used in documentation for data science packages
  • Collapse of in-page navigation not optimal

Other factors

• Furo SymPy prototype gets excellent Lighthouse scores before any customization:
  • Performance: 100
  • Accessibility: 98
  • Best Practices: 100
  • SEO (search engine optimization): 90
• Furo is well supported, having frequent updates
• Furo's developer is very accessible, even commenting on a SymPy thread without our asking them
• "Pretty much every color on the screen is customizable" per Furo's developer so we should be able to address what respondents disliked about colors
• Furo is the only theme that shows a fully expanded table of contents on the right sidebar
• Furo was recommended by Joannah Nanjekye, who spent much time working on the documentation for 2021 Google Season of Docs

Recommendation

For the above reasons, we should proceed with Furo as the new Sphinx theme. Customizing the theme should address some of the deficits of the prototype, such as colors.

Should there be some unexpected reason we cannot customize Furo as desired, we could try PyData or Book.

Other comments from respondents

Finally, nine people responded to "Are there any other comments you'd like to make?". Here is a summary of some things that stood out.

• All four options are good.
• Whichever theme you go with it’ll be an improvement.
• Please use that nice dark mode

Appendix: All comments from respondents

For the sake of completeness, all comments are shown below.

def print_answers(col):
    i = 1
    for e, v in df[[experience_level,col]].iloc:
        if pandas.isna(v):
            continue
        print(f"{i}. {v.strip()} (experience level: {int(e) if not pandas.isna(e) else 'N/A'})\n")
        i += 1

tag = "---"
for name, values in df.iteritems():
    if "like" in str(name).lower():
        print(tag + " " + name + " " + tag)
        print_answers(name)

--- What do you LIKE about the PyData theme? ---
1. search easy to find, updating position in doc on right (experience level: 5)

2. It is a neat theme (experience level: 4)

3. Clear, succint, very little clutter (experience level: 5)

4. I like the clean layout. (experience level: 4)

5. It is neat and simple, with left and right sidebars proving useful. The top sidebar is also quite convenient since scrolling on the left sidebar would be reduced as opposed to themes which lack the top sidebar. (experience level: 3)

6. clear view without distractions (experience level: 2)

7. Flat design, style uniformity accros sur the data exosystem, maintainability of a shared product (experience level: N/A)

8. The docstring rendering is easier to read than the old docs page. Better colors and less clutter. Fits on screen better. (experience level: 3)

9. Readable, not a huge shift (experience level: 3)

10. Desktop: Clean look. The categories across the top. The search box in a fixed position on the left navbar. Good highlighting format (for ?highlight=xxx in URL). Good differentiation between regular text, links (bold, blue), and code (pink). Good permalink setup: ¶ appears when link text moused over, then ¶ highlights when moused over. Code blocks are clearly delineated (using gray box) and their background is the same as rest of page. Easy to triple-click to select a line of code from a code block. Phone: Fairly easy to copy code from a code block. (experience level: 3)

11. It's very clean looking. Easy to navigate. (experience level: 3)

12. It looks neat and has a floating content menu. (experience level: 5)

13. Reminds me of scipy (experience level: 4)

14. it's clean (experience level: 4)

--- What do you DISLIKE about the PyData theme? ---
1. Looks kinda incomplete (experience level: 4)

2. 3 levels of nesting (top, left, right) was not instantly obvious for how to use. (experience level: 5)

3. I am bothered by the way useful links switch from side-to-side. I do understand the the links at right are to anchors on the particular page. However, sometimes having links on one side, but not the other did not seem natural and was initially confusing. (experience level: 4)

4. Left sidebar occupies a lot of space on the screen. (experience level: 3)

5. nothing (experience level: 2)

6. The first thing I did was click "API ref" in the right side menu, but it didn't take me to a new page. I would expect selecting from a menu on the right would then show new content about that topic.

It needs more sympy colors to feel like sympy's documentation.

The front page is quite boring with only four big headers. If you look at matplotlib, numpy, pandas, etc. the front page is very engaging with the graphics and big buttons.

You have to click too many times to drill down to seeing an actual page with information on it. In the old docs you have a big ToC on the front page and one click usually gets you to useful information.

Most pages feel too short. Web browsers can scroll (infinitely). It's preferable to make longer pages with hyperlink targets and table of contents menus. (experience level: 3)

7. White background, black text. Could use more links or tables of contents. (experience level: 3)

8. Desktop: In left navbar, not super-obvious which item this page is (highlighting not that strong). Distinction between h2 and h3 not that obvious, especially when h2 text is lowercase and h3 text is uppercase. Phone: Initial view of many pages (home, search) wastes a lot of vertical space. When tap a link in tree at top of page, there's no visible indication that the page content changed because link tree takes up entire screen height. (experience level: 3)

9. I don't like the right-hand TOC that only expands the section you're currently in. (experience level: 3)

10. Nothing in particular but if possible there should be an option for dark mode. (experience level: 5)

11. Doesn't remind me of sympy (experience level: 4)

12. content is too narrow (experience level: 4)

--- What do you LIKE about the Book theme? ---
1. Simple and minimalistic (experience level: 4)

2. Clean, intuitive (experience level: 5)

3. Good clean styling. Accordion navigation at left. Easy to find way to collapse left hand navigation pane. (experience level: 4)

4. Collapsible sidebar and full screen mode would help to view docs better once users find what they are looking for. (experience level: 3)

5. clean view (experience level: 2)

6. I like the font (experience level: 3)

7. Desktop: Clear differentiation of headings from regular text, and h2 from h3; and current section in left navbar; due to blue color. Clear differentiation of function syntax and parameters due to background color. Permalinks nicely handled: ¶ appears when mouse over link text, then gets darker when mouse over ¶. Phone: Pretty good use of vertical space. Page content is front and center, and nicely presented when go to a new page. Previous and Next links at bottom of page give visitor suggestions of where to go next. (experience level: 3)

8. Again, very clean and easy to navigate. Works well on my phone, too. (experience level: 3)

9. it's appealing to read (experience level: 4)

--- What do you DISLIKE about the Book theme? ---
1. I don't like the expandable index as much as I like the top selection of doc + left index; I also find the larger logo overpowering of the important element (like search and index). (experience level: 5)

2. top bar, that menu button, full screen toggle and download button will hardly ever be used, yet they consume precious vertical space. (experience level: 5)

3. SymPy logo takes up a lot of space on every page. Code indents seem off in some pages. (experience level: 3)

4. top controls distract me ) (experience level: 2)

5. I can't really tell the difference in this one and the first one I was shown. So all the same comments apply. (experience level: 3)

6. White background (experience level: 3)

7. Phone: Icons at top right (fullscreen and download) unlikely to be used often, and draw visitor's attention. No index (link tree) for within current page. No constant reminder of which site I'm on--could we add "SymPy" or the logo to the header, between hamburger menu and two icons at top right? (experience level: 3)

8. Same as pydata: I don't like the right-hand TOC that only expands the section you're currently in. (experience level: 3)

9. the contest has got to be really organized in this format (experience level: 4)

--- What do you LIKE about the Furo theme? ---
1. The sidebar for contents is pretty cool (experience level: 4)

2. Clean, intuitive and puts all vertical space to use. (experience level: 5)

3. Almost as good as book. Clean theme, easy to navigate. (experience level: 4)

4. The inbuilt dark mode could be useful. (experience level: 3)

5. I like that right and left menus are collapsable. Many times I checked Sympy documentation using my tablet and I like to have a bigger are dedicated to the document I’m reading. (experience level: 1)

6. nothing (experience level: 2)

7. The dark theme button is right here, I find this layout very easy on the eye (experience level: N/A)

8. Black background, clear font, THIS IS THE ONE (experience level: 3)

9. Desktop: Clear differentiation between headings and regular text, and h2 from h3. Pretty clear which is current section on left navbar due to bolding. Permalink setup pretty good: ¶ appears when mouse over link text, cursor turns to hand when mouse over ¶. Phone: Having two hamburger menus, one for site tree and one for sections-on-this-page tree. Always displays site title at top. (experience level: 3)

10. * I like the right-hand TOC where you always see all sections. For me, this is much better than the ones (book and pydata) where it only expands the section you're currently in.

* I think this one just "looks" the best overall. (experience level: 3)

11. As before, clean and easy to navigate. Also, I like the light and dark theme options. Easy on the eyes. (experience level: 3)

12. In my opinion it is the perfect theme. (experience level: 5)

13. It's dark. I like dark (experience level: 4)

14. looks nice (experience level: 4)

--- What do you DISLIKE about the Furo theme? ---
1. bold/highlighting seems a little cartoonish; I prefer a cleaner style (experience level: 5)

2. The lack of a secondary color makes it look unfinished (experience level: 4)

3. Nothing in particular (experience level: 5)

4. Does not collapse in-page navigation as well as 'book'. Also not as much easy changing of the view. (experience level: 4)

5. SymPy logo takes up a lot of space on every page. (experience level: 3)

6. too dark without sufficient contrast (experience level: 2)

7. That it’s not more widely used in data science docs (experience level: N/A)

8. Same comment as last. Looks practically the same as the prior two. Same comments apply. (experience level: 3)

9. Nothing (experience level: 3)

10. Desktop: Not a big fan of background color in code blocks; could be improved by making the color lighter (less saturated). Phone: Bit of an "interference effect" when scroll vertically as top title bar hides (?) page content. (experience level: 3)

11. colors are distracting (experience level: 4)

--- What do you LIKE about the Read the Docs (RTD) theme? ---
1. I like the "stickies" like notes on the page and that IDE examples seem less intrusive (experience level: 5)

2. The colors are nice (experience level: 4)

3. Conventional, i.e. familiar to many users (experience level: 5)

4. Very familiar interface as this is used by lots of standard documentation. Easy to navigate. All the quicklinks on one side. (experience level: 4)

5. User friendly interface. Familiar theme, therefore could attract many users. (experience level: 3)

6. bright and colorfull (experience level: 2)

7. Simple, effective (experience level: N/A)

8. I like this one the best! The menu on the left works as expected. You click and it takes you to a new page of info. And then the menu expands letting you quickly navigate to new relevant content. I think if that big blue square in the top left was sympy green (and probably other blues need to be swapped to greens) then we have a winner! (experience level: 3)

9. Ehh (experience level: 3)

10. Desktop: Headings are nice and bold. Clear differentiation of h2 from h3. Using increasingly opaque colors in left navbar demonstrates where in site tree visitor is. Permalinks pretty good: ¶ appears when mouse over link text, cursor changes when mouse over ¶. Phone: . (experience level: 3)

11. Nice and familiar. (experience level: 3)

--- What do you DISLIKE about the Read the Docs (RTD) theme? ---
1. I kind of like to know where the Next and Previous buttons are taking me. (experience level: 5)

2. I'd like the side bar to be collapsible (experience level: 4)

3. One cannot expand subsections of the Reference documentation directly to the left, but is forced to click the links in the body. Easier to get lost on big pages since it lacks the "on-page-navigation" bar to the right. (experience level: 5)

4. Does not have the easy view customization of 'book' (experience level: 4)

5. Lack of right sidebar to filter topics on a particular page. (experience level: 3)

6. I find the other three themes clearer. (experience level: 1)

7. colours distract from text (experience level: 2)

8. Seems devoid of personality, this theme is far too popular for its own good (experience level: N/A)

9. Needs green in colors. Needs more engaging front page. There is a lot of prose style docs (often module docstring) in the API. That really needs to be move into one of the other big 4 groups of the documentation. The api should strictly be the docstrings of classes, functions, etc. (experience level: 3)

10. It’s not the fur theme (experience level: 3)

11. Previous and Next links don't display what those page titles are (until visitor mouses over). Desktop: Not a big fan of color backgrounds of left navbar and especially to the right of page content; could be improved by changing colors. Determining which page vistior is on, in left navbar, could be easier: highlighting of current page is link a little weak. Phone: Very much dislike how left navbar pushes page content to the right; prefer overlay style of other themes. Very much dislike that there's no way to navigate to another page, or jump to another section of this page, after visitor scrolls down, because the hamburger menu isn't sticky. (experience level: 3)

12. left-hand TOC has issues

e.g. Try going to "Reference Documentation" > "Topics". Now under "Topics" you see the same sections as "Reference Documentation" repeated again. Now click "Topics" in here, and you actually get into "Topics". (experience level: 3)

13. Again it looks like any other python project website. It doesn't stand out in particular. (experience level: 5)

14. too old fashioned (experience level: 4)

--- Are there any other comments you'd like to make? ---
1. Thanks for organizing this! (experience level: 5)

2. None in particular, looks great! (experience level: 5)

3. Book is cleaner looking and allows the viewer more adaptations of the display. However, it is likely to be less familiar and a little harder to navigate than rtd for most people. I think either would be acceptable. Book seems a little more 'modern'. The 'modernity' will probably only last for a couple of years as styles change so rapidly. (experience level: 4)

4. All four options are good. (experience level: 1)

5. thank you for your work on SymPy. (experience level: 2)

6. Whichever theme you go with it’ll be an improvement. Thank you for taking the time to implement this and to ask for feedback ! (experience level: N/A)

7. The first 3 all look the same and have poorly designed menus that don't function as expected. It also seems that there are two menus on some, which is confusing. The last one RTD is really the only one that looks functional to me. (experience level: 3)

8. Please use that nice dark mode (experience level: 3)

9. PyData, Book, and Furo would all be good choices.
Category names (SymPy Tutorial, SymPy Guides, Explanation, Reference Documentation, Miscellaneous) should be shortened, for example remove "SymPy", change "Reference Documentation" to API(?). (experience level: 3)