Managing a Mainstream Open Source Library is Tough

1 minute read

Managing a mainstream open-source library is a full-time job that requires implementing an efficient workflow to avoid being overwhelmed.

This is particularly noticeable with Langchain, where at certain times of the day, you refresh the Pull Requests page on GitHub, and there's almost a new one every 10 minutes.

While I'm not a fan of all Langchain's features, I'm quite admiring of the daily workload of the dev teams: nearly 10,000 Pull Requests processed to date for ~2,400 contributors. To handle this flow of code to review, they have implemented the following workflow:

To contribute, you must follow the standard "fork and pull request" procedure, meaning you cannot push directly to their repo; you must make a PR from a fork you have modified.

Before coding anything, you must install the project and its various dependencies with poetry, which will also create a virtual environment for you and centralize all project configurations in the pyproject.toml file.

If you implement a new feature, you will need to document it, add unit tests (and integration tests if necessary), and provide a Jupyter notebook to facilitate understanding and provide concrete usage examples.

Then your code will have to pass a battery of tests, including the following technologies:

  • pytest for unit tests / integration tests.

  • ruff, a super-fast linter / code formatter written in Rust.

  • mypy, which will ensure that all the typing of your functions is consistent.

  • codespell will search for any typos that may have crept into your comments / docstrings.

(Incidentally, it's interesting to see that they don't use GitHub pre-commit).

Makefile is used to automate the commands to be executed, so you can run "make test", "make format", "make lint", etc., locally to check your code.

Finally, when you push your PR, a GitHub Actions CI will trigger to perform the same checks as mentioned above, but additionally, it will ensure code compatibility with Python versions 3.8, 3.9, 3.10, and 3.11.

If your proposal passes all tests and is deemed worthy of interest, it will be reviewed by a team member and potentially merged.

This will undoubtedly put off more than one person, but the idea is to guarantee and maintain a high level of quality. This approach will allow filtering, simplify the review, and decrease the time spent on it.

There are undoubtedly some good practices to be gained.


You May Also Enjoy