3mo ago

Python Performance: Why 'if not list' is 2x Faster Than Using len()

blog.codingconfessions.com

Lobste.rs @lemmy.bestiver.se

RSS Bot @lemmy.bestiver.se

BOT

4mo ago

Python Performance: Why 'if not list' is 2x Faster Than Using len()

blog.codingconfessions.com /p/python-performance-why-if-not-list

Technology @lemmy.world

abhi9u @lemmy.world

4mo ago

Python Performance: Why 'if not list' is 2x Faster Than Using len()

blog.codingconfessions.com /p/python-performance-why-if-not-list

You're viewing a single thread.

53 comments

I think the explicitness of checking length is worth the performance cost. If ur writing code for speed ur not using python.
- I'd argue that if it's strict explicitness you want, python is the wrong language. if not var is a standard pattern in python. You would be making your code slower for no good reason.
  
  You always want explicitness when programming. Not everyone reading your code will be deep into Python and relying on falsiness makes it harder to understand.
  
  While I agree to some extent, if not var is more than clear enough for anyone that knows python. If that pattern confuses someone, they probably aren't at level where they should be dealing with python at a professional level. The same way you would expect people to understand pointers and references before delving into C code.
  This sort of stuff is something I taught first year engineering student in their programming introductory course, it's just how python is written.
  For what it's worth, it's sort of similar in Erlang/Elixir. If you want to check if a list is empty, checking the length of the list is discouraged. Instead you would use Enum.empty?().
  
  Well, it certainly wouldn't be the first time that I call some code unnecessarily hard to read and others call it pythonic.
  I understand that truthiness has an advantage when you don't have static types, because it deals somewhat reasonably with most types, and then that is why many experienced Python programmers tend to use it. But I maintain the position that it therefore necessarily also makes it harder to read the code, because you necessarily provide the reader with fewer hints as to what is actually in that variable. It is very much like code using lots of generics in statically typed code.
  As for if Enum.empty?(var), I actually prefer that to checking the length. That syntax in particular, I find a bit too complex – my preferred version is if var.is_empty() – but I still think it makes it easier to read than a length check.
  Of course, there's the nuance that it's more syntax to remember for actually writing the code. And in particular in dynamically typed languages, your editor may not be able to auto-complete that, so I can understand just not bothering with the extra syntax and doing len == 0 instead. It's fine.
  
  you necessarily provide the reader with fewer hints as to what is actually in that variable
  Then make it explicit. I much prefer this:
  
  def do_foo(bar: list | None): if not bar: return ...
  This one communicates to me that the function only makes sense with a non-empty list.
  To this:
  
  def do_foo(bar): if len(bar) == 0: return
  This just tells me bar is something that has a length (list, dict, str, etc).
  And this is way worse:
  
  def do_foo(bar: list | None): if len(bar) == 0: return
  This tells me we want an exception if bar is None, which I think is bad style, given that we're explicitly allowing None here. If we remove the | None and get an exception, than the code is broken because I shouldn't be able to get that value in that context.
  If I care about the difference between None and empty, then I'll make that explicit:
  
  if bar is None: ... if not bar: ...
  In any case, I just do not like if len(bar) == 0 because that looks like a mistake (i.e. did the dev know it could throw an error if it's not a list? Was that intentional?).
  
  Containers being "truthy" is quite basic Python and you will find this idiom used in nearly every Python code base in my experience
  
  Yeah, I'm talking less deep than that. Plenty programming beginners will be reading Python code. And personally, I'm a fulltime software engineer, but just don't do much Python, so while I had it in the back of my mind that Python does truthiness, I would have still thought that var must be a boolean, because it's being negated. Obviously, a different variable name might've given me more of a clue, but it really doesn't reduce mental complexity when I can't be sure what's actually in a variable.
  
  But if those beginners want to stop being beginners, then they must learn the basics of the language. It makes no more sense to demand that everyone who programs in Python caters to beginners, than it makes to demand that everyone writing in English write at a 3rd grade reading level for the sake of English language learners
  
  I think you just like the truth test. Frankly I have used Python for over 25 years and have been doing programming for almost 50 years in many different languages. If you think I am somehow a beginner, I would disagree. The truth test is just like so many other Python specific idioms that grow in number by the year. They are not at all obvious unless you are deep into Python. Moreover, the truth test and the len() test are not the same test. One might be able to use either in a specific case, but that is case specific and which is more readable is up to the developer and we may well disagree on that choice. The other consideration in Python, speed of writing code which is often why many of us use Python and that may lead to different choices too including based on habit.
  Lot of this reminds me of the Pascal vs. C debate of the 1980's. Pascal was all about readability over compactness. C on the other hand, seemed to attract people that loved to write very compact code that was almost impossible to figure out on first glance. Me personally, I guess I'd choose C over Pascal but write the C in more of a Pascal authoring philosophy. Similarly, with Python, I often do not go for all of the Python idioms. Lot of that is just writing what I am thinking, and the rest is probably habit. If I am going to test 0 length then I'll probably test zero length. I do not find it at all obvious that, oh, I want to test 0 length so the Python idiom for that is to truth test. I absolutely know that to be the case on certain types of objects, but it probably is not going to be my first choice.
  
  Yeah, my stance for both of those is the same: If the complexity aids you in communicating better, then use it. But if you're using big words where small words would do, then you're doing a disservice to your readers.
- I never understood that argument. If you can be sure the type is a collection (and this you always should) not list is so moch easier to read and understood than the length check.
  
  How many elements in that list? Ah, it's not list. It's list, of course, we checked. But it's not.
  
  Compact does not mean easier to understand. They are different tests. The point is, do not make code less readable for speed unless speed matters. I am not going to argue which is more readable in any specific case. That is up to the developer.
  
  Is it? Why introduce an additional conversion from not list means empty list that u have to hold in your head. I want to check length I check length I would argue is easyer to comprehend.
  
  Because that's a fundamental aspect of Python. When you're using a language, you should be familiar with the truthiness values. In Python, it's pretty sane:
  [], {}, set(), "", None, False 0 and related values are all "falesy"
  everything else is truthy
  Basically, if you have non-default values, it's truthy. Why wouldn't you trust basic features of the language?
  
  Because I have to do the is this falsy to what I'm actually interested conversion in my head.
  Say ur deep inside some complicated piece of logic and u are trying to understand. Now u have a bunch of assumptions in your head. You should be trying to eliminate as many if these assumptions with good code as possible eg u test ur fail case and return/continue that so u don't need to keep that assumption in ur head.
  Say I then come along a if not x then you have to figure out what is x what is the truthiness of its type. If I come across an if len(x) == 0 then I automatically know that x is some collection of objects and I'm testing its emptiness.
  
  That's why there's type hinting, unit tests, and doc strings. I don't need to guess what the type is intended to be, I can see it.
  
  But that's an extra step of logic u must hold in ur head while trying to understand 12 other things.
  
  What's the extra logic?
  python
  
  if x:
  This always evaluates to True if it's non-empty. There's no extra logic.
  If you have to keep 12 things in your head, your code is poorly structured/documented. A given function should be simple, making it plainly obvious what it's intended to do. Use type hints to specify what a variable should be, and use static analysis to catch most deviations. The more you trust your tools, the more assumptions you can safely make.
- Why? not x means x is None or len(x) == 0 for lists. len(x) == 0 will raise an exception if x is None. In most cases, the distinction between None and [] isn't important, and if it is, I'd expect separate checks for those (again, for explicitness) since you'd presumably handle each case differently.
  In short:
  if the distinction between None and [] is important, have separate checks
  if not, not x should be your default, since that way it's a common pattern for all types
  
  I try to avoid having the same variable with different types I find it is often the cause of difficult to debug bugs. I struggle to think of a case where u would be performing a check that could be an empty list or None where both are expected possible values.
  
  Really? I get that all the time. I do web dev, and our APIs have a lot of optional fields.
  
  I do web dev
  Theirs ur problem.
  But in all seriousness I think if u def some_func(*args, kwarg=[]) Is a more explicit form of def some_func(*args, kwarg=None)
  
  def some_func(*args, kwarg=[])
  Don't do this:
  python
  
  def fun(l=[]): l.append(len(l)) return l fun() # [0] fun() # [0, 1] fun(l=[]) # [0] fun() # [0, 1, 2] fun(l=None) # raise AttributeError or TypeError if len(l) comes first
  This can be downright cryptic if you're passing things dynamically, such as:
  python
  
  def caller(*args, **kwargs): fun(*args, **kwargs)
  It's much safer to do a simple check at the beginning:
  python
  
  if not l: l = []
  
  I like the exception being raised their is no reason I should be passing in None to the function it means I've fucked up the value of whatever I'm passing in at some point.
  
  Then make it explicit:
  python
  
  if l is None: raise ValueError("Must provide a valid value for...")
  Having an attribute or type error rarely provides the right amount of context to immediately recognize the error, especially if it's deep inside the application. A lot of our old code makes stupid errors like TypeError: operator - not defined on types NoneType and float, because someone screwed up somewhere and wasn't strict on checks. Don't reply on implicit exceptions, explicitly raise them so you can add context, because sometimes stacktraces get lost and all you have is the error message.
  But in my experience, the practical difference between [] and None is essentially zero, except in a few cases, and those should stand out. I have a few places with logic like this:
  python
  
  if l is None: raise MyCustomInvalidException("Must provide a list") if not l: # nothing to do return
  For example, if I make a task runner, an empty list could validly mean no arguments, while a null list means the caller screwed up somewhere and probably forgot to provide them.
  Explicit is better than implicit, and simple is better than complex.
  
  Oh no a stray None! Take cover ...
  Robust codebase should never fail from a stray None
  Chaos testing is specifically geared towards bullet proofing code against unexpected param types including None.
  The only exception is for private support function for type specific checking functions. Where it's obviously only for one type ever.
  We live in clownworld, i'm a clown and keep the company of shit throwing monkeys.
  
  Ur function args if fucked up should always throw an error that's the entire point of python type hints
  
  type hints are static, not necessarily runtime.
  A chaos monkey throws everything at everything to see what breaks.
  That won't be caught by perfect type hints, which is merely one tool in the toolbox.
  and when things break, often hear WAD, works as designed. Or some other nonsense excuse.
- In complex cases where speed is less important than maintainability, I tend to agree.
  In this case, a simple comment would suffice. And in fact nothing at all would be okay for any half-competent Python coder, as testing if lists are empty with if not is super-standard.
- Another use case is to reduce maintenance cost

53 comments