Best practices for naming python utils / extending core modules?

Question

So a lot of the time my utils end up with a structure that mirrors the core library. I might end up writing a multiline version of str.center, an itertools-y function that returns the first or last element of an iterator that matches a predicate, a datetime function that returns the end of the current day, maybe a custom json encoder to extend json.dumps/loads.

I could reexport these from modules that have the same name as the core modules: myproject.utils.string, myproject.utils.itertools, myproject.utils.datetime, and, if I reexport * from the original module, I can replace import json with import myproject.utils.json. (This is what, for example, Flask does with its json module.) But that feels like a recipe for confusion.

Alternatively, I could give them a name based on the original modules, maybe xstring, xitertools, (in retrospect, this seems gross if not adult-rated).

Are there better options? What would best practices be?

Answer 1

Very rarely, I'll add a module that re-exports various utilities I'll need throughout the project, so that in every file I can do

from .prelude import *

and get all those utilities or definitions in scope. In the same vein, it can make sense to define utils modules that are used throughout the code. I'd suggest names like .utils.foo or .foo_utils since they make it clear what's going on, whereas names like .xfoo might be a bit less obvious. Whether these utility modules end up extending a standard library module should be a case by case decision. I think it will usually be clearer if the extensions are separate (it's just going to be one extra import statement per use), but in cases like Flask there's a good argument to be made to offer users a unified interface so that they no longer have to use the standard library module. In this specific case, Flask has to do this in order to provide different defaults to the various JSON reading/writing functions.

But my experience is that a lot of utilities are not so cross-cutting that they should be extracted into an utils module. Often, these utility functions will only be used by a single module, and can then be defined within that module. Since Python has no real encapsulation, this won't affect testability.

It is also potentially an anti-pattern to think in terms of “utils” modules. If this code is useful for the project, then it's worth putting into a module under a name that clearly conveys the role of that code in the project. For example, the code in myproject.utils.json might actually relate to reading/writing a particular file format, e.g. myproject.expense_report. A function to center multiline strings might be better presented as part of myproject.text_layout than myproject.utils.string.