Shortcuts on this page
r m x toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
1# orm/collections.py
2# Copyright (C) 2005-2024 the SQLAlchemy authors and contributors
3# <see AUTHORS file>
4#
5# This module is part of SQLAlchemy and is released under
6# the MIT License: https://www.opensource.org/licenses/mit-license.php
8"""Support for collections of mapped entities.
10The collections package supplies the machinery used to inform the ORM of
11collection membership changes. An instrumentation via decoration approach is
12used, allowing arbitrary types (including built-ins) to be used as entity
13collections without requiring inheritance from a base class.
15Instrumentation decoration relays membership change events to the
16:class:`.CollectionAttributeImpl` that is currently managing the collection.
17The decorators observe function call arguments and return values, tracking
18entities entering or leaving the collection. Two decorator approaches are
19provided. One is a bundle of generic decorators that map function arguments
20and return values to events::
22 from sqlalchemy.orm.collections import collection
23 class MyClass(object):
24 # ...
26 @collection.adds(1)
27 def store(self, item):
28 self.data.append(item)
30 @collection.removes_return()
31 def pop(self):
32 return self.data.pop()
35The second approach is a bundle of targeted decorators that wrap appropriate
36append and remove notifiers around the mutation methods present in the
37standard Python ``list``, ``set`` and ``dict`` interfaces. These could be
38specified in terms of generic decorator recipes, but are instead hand-tooled
39for increased efficiency. The targeted decorators occasionally implement
40adapter-like behavior, such as mapping bulk-set methods (``extend``,
41``update``, ``__setslice__``, etc.) into the series of atomic mutation events
42that the ORM requires.
44The targeted decorators are used internally for automatic instrumentation of
45entity collection classes. Every collection class goes through a
46transformation process roughly like so:
481. If the class is a built-in, substitute a trivial sub-class
492. Is this class already instrumented?
503. Add in generic decorators
514. Sniff out the collection interface through duck-typing
525. Add targeted decoration to any undecorated interface method
54This process modifies the class at runtime, decorating methods and adding some
55bookkeeping properties. This isn't possible (or desirable) for built-in
56classes like ``list``, so trivial sub-classes are substituted to hold
57decoration::
59 class InstrumentedList(list):
60 pass
62Collection classes can be specified in ``relationship(collection_class=)`` as
63types or a function that returns an instance. Collection classes are
64inspected and instrumented during the mapper compilation phase. The
65collection_class callable will be executed once to produce a specimen
66instance, and the type of that specimen will be instrumented. Functions that
67return built-in types like ``lists`` will be adapted to produce instrumented
68instances.
70When extending a known type like ``list``, additional decorations are not
71generally not needed. Odds are, the extension method will delegate to a
72method that's already instrumented. For example::
74 class QueueIsh(list):
75 def push(self, item):
76 self.append(item)
77 def shift(self):
78 return self.pop(0)
80There's no need to decorate these methods. ``append`` and ``pop`` are already
81instrumented as part of the ``list`` interface. Decorating them would fire
82duplicate events, which should be avoided.
84The targeted decoration tries not to rely on other methods in the underlying
85collection class, but some are unavoidable. Many depend on 'read' methods
86being present to properly instrument a 'write', for example, ``__setitem__``
87needs ``__getitem__``. "Bulk" methods like ``update`` and ``extend`` may also
88reimplemented in terms of atomic appends and removes, so the ``extend``
89decoration will actually perform many ``append`` operations and not call the
90underlying method at all.
92Tight control over bulk operation and the firing of events is also possible by
93implementing the instrumentation internally in your methods. The basic
94instrumentation package works under the general assumption that collection
95mutation will not raise unusual exceptions. If you want to closely
96orchestrate append and remove events with exception management, internal
97instrumentation may be the answer. Within your method,
98``collection_adapter(self)`` will retrieve an object that you can use for
99explicit control over triggering append and remove events.
101The owning object and :class:`.CollectionAttributeImpl` are also reachable
102through the adapter, allowing for some very sophisticated behavior.
104"""
106import operator
107import weakref
109from sqlalchemy.util.compat import inspect_getfullargspec
110from . import base
111from .. import exc as sa_exc
112from .. import util
113from ..sql import coercions
114from ..sql import expression
115from ..sql import roles
117__all__ = [
118 "collection",
119 "collection_adapter",
120 "mapped_collection",
121 "column_mapped_collection",
122 "attribute_mapped_collection",
123]
125__instrumentation_mutex = util.threading.Lock()
128class _PlainColumnGetter(object):
129 """Plain column getter, stores collection of Column objects
130 directly.
132 Serializes to a :class:`._SerializableColumnGetterV2`
133 which has more expensive __call__() performance
134 and some rare caveats.
136 """
138 def __init__(self, cols):
139 self.cols = cols
140 self.composite = len(cols) > 1
142 def __reduce__(self):
143 return _SerializableColumnGetterV2._reduce_from_cols(self.cols)
145 def _cols(self, mapper):
146 return self.cols
148 def __call__(self, value):
149 state = base.instance_state(value)
150 m = base._state_mapper(state)
152 key = [
153 m._get_state_attr_by_column(state, state.dict, col)
154 for col in self._cols(m)
155 ]
157 if self.composite:
158 return tuple(key)
159 else:
160 return key[0]
163class _SerializableColumnGetter(object):
164 """Column-based getter used in version 0.7.6 only.
166 Remains here for pickle compatibility with 0.7.6.
168 """
170 def __init__(self, colkeys):
171 self.colkeys = colkeys
172 self.composite = len(colkeys) > 1
174 def __reduce__(self):
175 return _SerializableColumnGetter, (self.colkeys,)
177 def __call__(self, value):
178 state = base.instance_state(value)
179 m = base._state_mapper(state)
180 key = [
181 m._get_state_attr_by_column(
182 state, state.dict, m.mapped_table.columns[k]
183 )
184 for k in self.colkeys
185 ]
186 if self.composite:
187 return tuple(key)
188 else:
189 return key[0]
192class _SerializableColumnGetterV2(_PlainColumnGetter):
193 """Updated serializable getter which deals with
194 multi-table mapped classes.
196 Two extremely unusual cases are not supported.
197 Mappings which have tables across multiple metadata
198 objects, or which are mapped to non-Table selectables
199 linked across inheriting mappers may fail to function
200 here.
202 """
204 def __init__(self, colkeys):
205 self.colkeys = colkeys
206 self.composite = len(colkeys) > 1
208 def __reduce__(self):
209 return self.__class__, (self.colkeys,)
211 @classmethod
212 def _reduce_from_cols(cls, cols):
213 def _table_key(c):
214 if not isinstance(c.table, expression.TableClause):
215 return None
216 else:
217 return c.table.key
219 colkeys = [(c.key, _table_key(c)) for c in cols]
220 return _SerializableColumnGetterV2, (colkeys,)
222 def _cols(self, mapper):
223 cols = []
224 metadata = getattr(mapper.local_table, "metadata", None)
225 for (ckey, tkey) in self.colkeys:
226 if tkey is None or metadata is None or tkey not in metadata:
227 cols.append(mapper.local_table.c[ckey])
228 else:
229 cols.append(metadata.tables[tkey].c[ckey])
230 return cols
233def column_mapped_collection(mapping_spec):
234 """A dictionary-based collection type with column-based keying.
236 Returns a :class:`.MappedCollection` factory with a keying function
237 generated from mapping_spec, which may be a Column or a sequence
238 of Columns.
240 The key value must be immutable for the lifetime of the object. You
241 can not, for example, map on foreign key values if those key values will
242 change during the session, i.e. from None to a database-assigned integer
243 after a session flush.
245 """
246 cols = [
247 coercions.expect(roles.ColumnArgumentRole, q, argname="mapping_spec")
248 for q in util.to_list(mapping_spec)
249 ]
250 keyfunc = _PlainColumnGetter(cols)
251 return lambda: MappedCollection(keyfunc)
254class _SerializableAttrGetter(object):
255 def __init__(self, name):
256 self.name = name
257 self.getter = operator.attrgetter(name)
259 def __call__(self, target):
260 return self.getter(target)
262 def __reduce__(self):
263 return _SerializableAttrGetter, (self.name,)
266def attribute_mapped_collection(attr_name):
267 """A dictionary-based collection type with attribute-based keying.
269 Returns a :class:`.MappedCollection` factory with a keying based on the
270 'attr_name' attribute of entities in the collection, where ``attr_name``
271 is the string name of the attribute.
273 .. warning:: the key value must be assigned to its final value
274 **before** it is accessed by the attribute mapped collection.
275 Additionally, changes to the key attribute are **not tracked**
276 automatically, which means the key in the dictionary is not
277 automatically synchronized with the key value on the target object
278 itself. See the section :ref:`key_collections_mutations`
279 for an example.
281 """
282 getter = _SerializableAttrGetter(attr_name)
283 return lambda: MappedCollection(getter)
286def mapped_collection(keyfunc):
287 """A dictionary-based collection type with arbitrary keying.
289 Returns a :class:`.MappedCollection` factory with a keying function
290 generated from keyfunc, a callable that takes an entity and returns a
291 key value.
293 The key value must be immutable for the lifetime of the object. You
294 can not, for example, map on foreign key values if those key values will
295 change during the session, i.e. from None to a database-assigned integer
296 after a session flush.
298 """
299 return lambda: MappedCollection(keyfunc)
302class collection(object):
303 """Decorators for entity collection classes.
305 The decorators fall into two groups: annotations and interception recipes.
307 The annotating decorators (appender, remover, iterator, converter,
308 internally_instrumented) indicate the method's purpose and take no
309 arguments. They are not written with parens::
311 @collection.appender
312 def append(self, append): ...
314 The recipe decorators all require parens, even those that take no
315 arguments::
317 @collection.adds('entity')
318 def insert(self, position, entity): ...
320 @collection.removes_return()
321 def popitem(self): ...
323 """
325 # Bundled as a class solely for ease of use: packaging, doc strings,
326 # importability.
328 @staticmethod
329 def appender(fn):
330 """Tag the method as the collection appender.
332 The appender method is called with one positional argument: the value
333 to append. The method will be automatically decorated with 'adds(1)'
334 if not already decorated::
336 @collection.appender
337 def add(self, append): ...
339 # or, equivalently
340 @collection.appender
341 @collection.adds(1)
342 def add(self, append): ...
344 # for mapping type, an 'append' may kick out a previous value
345 # that occupies that slot. consider d['a'] = 'foo'- any previous
346 # value in d['a'] is discarded.
347 @collection.appender
348 @collection.replaces(1)
349 def add(self, entity):
350 key = some_key_func(entity)
351 previous = None
352 if key in self:
353 previous = self[key]
354 self[key] = entity
355 return previous
357 If the value to append is not allowed in the collection, you may
358 raise an exception. Something to remember is that the appender
359 will be called for each object mapped by a database query. If the
360 database contains rows that violate your collection semantics, you
361 will need to get creative to fix the problem, as access via the
362 collection will not work.
364 If the appender method is internally instrumented, you must also
365 receive the keyword argument '_sa_initiator' and ensure its
366 promulgation to collection events.
368 """
369 fn._sa_instrument_role = "appender"
370 return fn
372 @staticmethod
373 def remover(fn):
374 """Tag the method as the collection remover.
376 The remover method is called with one positional argument: the value
377 to remove. The method will be automatically decorated with
378 :meth:`removes_return` if not already decorated::
380 @collection.remover
381 def zap(self, entity): ...
383 # or, equivalently
384 @collection.remover
385 @collection.removes_return()
386 def zap(self, ): ...
388 If the value to remove is not present in the collection, you may
389 raise an exception or return None to ignore the error.
391 If the remove method is internally instrumented, you must also
392 receive the keyword argument '_sa_initiator' and ensure its
393 promulgation to collection events.
395 """
396 fn._sa_instrument_role = "remover"
397 return fn
399 @staticmethod
400 def iterator(fn):
401 """Tag the method as the collection remover.
403 The iterator method is called with no arguments. It is expected to
404 return an iterator over all collection members::
406 @collection.iterator
407 def __iter__(self): ...
409 """
410 fn._sa_instrument_role = "iterator"
411 return fn
413 @staticmethod
414 def internally_instrumented(fn):
415 """Tag the method as instrumented.
417 This tag will prevent any decoration from being applied to the
418 method. Use this if you are orchestrating your own calls to
419 :func:`.collection_adapter` in one of the basic SQLAlchemy
420 interface methods, or to prevent an automatic ABC method
421 decoration from wrapping your implementation::
423 # normally an 'extend' method on a list-like class would be
424 # automatically intercepted and re-implemented in terms of
425 # SQLAlchemy events and append(). your implementation will
426 # never be called, unless:
427 @collection.internally_instrumented
428 def extend(self, items): ...
430 """
431 fn._sa_instrumented = True
432 return fn
434 @staticmethod
435 @util.deprecated(
436 "1.3",
437 "The :meth:`.collection.converter` handler is deprecated and will "
438 "be removed in a future release. Please refer to the "
439 ":class:`.AttributeEvents.bulk_replace` listener interface in "
440 "conjunction with the :func:`.event.listen` function.",
441 )
442 def converter(fn):
443 """Tag the method as the collection converter.
445 This optional method will be called when a collection is being
446 replaced entirely, as in::
448 myobj.acollection = [newvalue1, newvalue2]
450 The converter method will receive the object being assigned and should
451 return an iterable of values suitable for use by the ``appender``
452 method. A converter must not assign values or mutate the collection,
453 its sole job is to adapt the value the user provides into an iterable
454 of values for the ORM's use.
456 The default converter implementation will use duck-typing to do the
457 conversion. A dict-like collection will be convert into an iterable
458 of dictionary values, and other types will simply be iterated::
460 @collection.converter
461 def convert(self, other): ...
463 If the duck-typing of the object does not match the type of this
464 collection, a TypeError is raised.
466 Supply an implementation of this method if you want to expand the
467 range of possible types that can be assigned in bulk or perform
468 validation on the values about to be assigned.
470 """
471 fn._sa_instrument_role = "converter"
472 return fn
474 @staticmethod
475 def adds(arg):
476 """Mark the method as adding an entity to the collection.
478 Adds "add to collection" handling to the method. The decorator
479 argument indicates which method argument holds the SQLAlchemy-relevant
480 value. Arguments can be specified positionally (i.e. integer) or by
481 name::
483 @collection.adds(1)
484 def push(self, item): ...
486 @collection.adds('entity')
487 def do_stuff(self, thing, entity=None): ...
489 """
491 def decorator(fn):
492 fn._sa_instrument_before = ("fire_append_event", arg)
493 return fn
495 return decorator
497 @staticmethod
498 def replaces(arg):
499 """Mark the method as replacing an entity in the collection.
501 Adds "add to collection" and "remove from collection" handling to
502 the method. The decorator argument indicates which method argument
503 holds the SQLAlchemy-relevant value to be added, and return value, if
504 any will be considered the value to remove.
506 Arguments can be specified positionally (i.e. integer) or by name::
508 @collection.replaces(2)
509 def __setitem__(self, index, item): ...
511 """
513 def decorator(fn):
514 fn._sa_instrument_before = ("fire_append_event", arg)
515 fn._sa_instrument_after = "fire_remove_event"
516 return fn
518 return decorator
520 @staticmethod
521 def removes(arg):
522 """Mark the method as removing an entity in the collection.
524 Adds "remove from collection" handling to the method. The decorator
525 argument indicates which method argument holds the SQLAlchemy-relevant
526 value to be removed. Arguments can be specified positionally (i.e.
527 integer) or by name::
529 @collection.removes(1)
530 def zap(self, item): ...
532 For methods where the value to remove is not known at call-time, use
533 collection.removes_return.
535 """
537 def decorator(fn):
538 fn._sa_instrument_before = ("fire_remove_event", arg)
539 return fn
541 return decorator
543 @staticmethod
544 def removes_return():
545 """Mark the method as removing an entity in the collection.
547 Adds "remove from collection" handling to the method. The return
548 value of the method, if any, is considered the value to remove. The
549 method arguments are not inspected::
551 @collection.removes_return()
552 def pop(self): ...
554 For methods where the value to remove is known at call-time, use
555 collection.remove.
557 """
559 def decorator(fn):
560 fn._sa_instrument_after = "fire_remove_event"
561 return fn
563 return decorator
566collection_adapter = operator.attrgetter("_sa_adapter")
567"""Fetch the :class:`.CollectionAdapter` for a collection."""
570class CollectionAdapter(object):
571 """Bridges between the ORM and arbitrary Python collections.
573 Proxies base-level collection operations (append, remove, iterate)
574 to the underlying Python collection, and emits add/remove events for
575 entities entering or leaving the collection.
577 The ORM uses :class:`.CollectionAdapter` exclusively for interaction with
578 entity collections.
581 """
583 __slots__ = (
584 "attr",
585 "_key",
586 "_data",
587 "owner_state",
588 "_converter",
589 "invalidated",
590 "empty",
591 )
593 def __init__(self, attr, owner_state, data):
594 self.attr = attr
595 self._key = attr.key
596 self._data = weakref.ref(data)
597 self.owner_state = owner_state
598 data._sa_adapter = self
599 self._converter = data._sa_converter
600 self.invalidated = False
601 self.empty = False
603 def _warn_invalidated(self):
604 util.warn("This collection has been invalidated.")
606 @property
607 def data(self):
608 "The entity collection being adapted."
609 return self._data()
611 @property
612 def _referenced_by_owner(self):
613 """return True if the owner state still refers to this collection.
615 This will return False within a bulk replace operation,
616 where this collection is the one being replaced.
618 """
619 return self.owner_state.dict[self._key] is self._data()
621 def bulk_appender(self):
622 return self._data()._sa_appender
624 def append_with_event(self, item, initiator=None):
625 """Add an entity to the collection, firing mutation events."""
627 self._data()._sa_appender(item, _sa_initiator=initiator)
629 def _set_empty(self, user_data):
630 assert (
631 not self.empty
632 ), "This collection adapter is already in the 'empty' state"
633 self.empty = True
634 self.owner_state._empty_collections[self._key] = user_data
636 def _reset_empty(self):
637 assert (
638 self.empty
639 ), "This collection adapter is not in the 'empty' state"
640 self.empty = False
641 self.owner_state.dict[
642 self._key
643 ] = self.owner_state._empty_collections.pop(self._key)
645 def _refuse_empty(self):
646 raise sa_exc.InvalidRequestError(
647 "This is a special 'empty' collection which cannot accommodate "
648 "internal mutation operations"
649 )
651 def append_without_event(self, item):
652 """Add or restore an entity to the collection, firing no events."""
654 if self.empty:
655 self._refuse_empty()
656 self._data()._sa_appender(item, _sa_initiator=False)
658 def append_multiple_without_event(self, items):
659 """Add or restore an entity to the collection, firing no events."""
660 if self.empty:
661 self._refuse_empty()
662 appender = self._data()._sa_appender
663 for item in items:
664 appender(item, _sa_initiator=False)
666 def bulk_remover(self):
667 return self._data()._sa_remover
669 def remove_with_event(self, item, initiator=None):
670 """Remove an entity from the collection, firing mutation events."""
671 self._data()._sa_remover(item, _sa_initiator=initiator)
673 def remove_without_event(self, item):
674 """Remove an entity from the collection, firing no events."""
675 if self.empty:
676 self._refuse_empty()
677 self._data()._sa_remover(item, _sa_initiator=False)
679 def clear_with_event(self, initiator=None):
680 """Empty the collection, firing a mutation event for each entity."""
682 if self.empty:
683 self._refuse_empty()
684 remover = self._data()._sa_remover
685 for item in list(self):
686 remover(item, _sa_initiator=initiator)
688 def clear_without_event(self):
689 """Empty the collection, firing no events."""
691 if self.empty:
692 self._refuse_empty()
693 remover = self._data()._sa_remover
694 for item in list(self):
695 remover(item, _sa_initiator=False)
697 def __iter__(self):
698 """Iterate over entities in the collection."""
700 return iter(self._data()._sa_iterator())
702 def __len__(self):
703 """Count entities in the collection."""
704 return len(list(self._data()._sa_iterator()))
706 def __bool__(self):
707 return True
709 __nonzero__ = __bool__
711 def fire_append_wo_mutation_event(self, item, initiator=None):
712 """Notify that a entity is entering the collection but is already
713 present.
716 Initiator is a token owned by the InstrumentedAttribute that
717 initiated the membership mutation, and should be left as None
718 unless you are passing along an initiator value from a chained
719 operation.
721 .. versionadded:: 1.4.15
723 """
724 if initiator is not False:
725 if self.invalidated:
726 self._warn_invalidated()
728 if self.empty:
729 self._reset_empty()
731 return self.attr.fire_append_wo_mutation_event(
732 self.owner_state, self.owner_state.dict, item, initiator
733 )
734 else:
735 return item
737 def fire_append_event(self, item, initiator=None):
738 """Notify that a entity has entered the collection.
740 Initiator is a token owned by the InstrumentedAttribute that
741 initiated the membership mutation, and should be left as None
742 unless you are passing along an initiator value from a chained
743 operation.
745 """
746 if initiator is not False:
747 if self.invalidated:
748 self._warn_invalidated()
750 if self.empty:
751 self._reset_empty()
753 return self.attr.fire_append_event(
754 self.owner_state, self.owner_state.dict, item, initiator
755 )
756 else:
757 return item
759 def fire_remove_event(self, item, initiator=None):
760 """Notify that a entity has been removed from the collection.
762 Initiator is the InstrumentedAttribute that initiated the membership
763 mutation, and should be left as None unless you are passing along
764 an initiator value from a chained operation.
766 """
767 if initiator is not False:
768 if self.invalidated:
769 self._warn_invalidated()
771 if self.empty:
772 self._reset_empty()
774 self.attr.fire_remove_event(
775 self.owner_state, self.owner_state.dict, item, initiator
776 )
778 def fire_pre_remove_event(self, initiator=None):
779 """Notify that an entity is about to be removed from the collection.
781 Only called if the entity cannot be removed after calling
782 fire_remove_event().
784 """
785 if self.invalidated:
786 self._warn_invalidated()
787 self.attr.fire_pre_remove_event(
788 self.owner_state, self.owner_state.dict, initiator=initiator
789 )
791 def __getstate__(self):
792 return {
793 "key": self._key,
794 "owner_state": self.owner_state,
795 "owner_cls": self.owner_state.class_,
796 "data": self.data,
797 "invalidated": self.invalidated,
798 "empty": self.empty,
799 }
801 def __setstate__(self, d):
802 self._key = d["key"]
803 self.owner_state = d["owner_state"]
804 self._data = weakref.ref(d["data"])
805 self._converter = d["data"]._sa_converter
806 d["data"]._sa_adapter = self
807 self.invalidated = d["invalidated"]
808 self.attr = getattr(d["owner_cls"], self._key).impl
809 self.empty = d.get("empty", False)
812def bulk_replace(values, existing_adapter, new_adapter, initiator=None):
813 """Load a new collection, firing events based on prior like membership.
815 Appends instances in ``values`` onto the ``new_adapter``. Events will be
816 fired for any instance not present in the ``existing_adapter``. Any
817 instances in ``existing_adapter`` not present in ``values`` will have
818 remove events fired upon them.
820 :param values: An iterable of collection member instances
822 :param existing_adapter: A :class:`.CollectionAdapter` of
823 instances to be replaced
825 :param new_adapter: An empty :class:`.CollectionAdapter`
826 to load with ``values``
829 """
831 assert isinstance(values, list)
833 idset = util.IdentitySet
834 existing_idset = idset(existing_adapter or ())
835 constants = existing_idset.intersection(values or ())
836 additions = idset(values or ()).difference(constants)
837 removals = existing_idset.difference(constants)
839 appender = new_adapter.bulk_appender()
841 for member in values or ():
842 if member in additions:
843 appender(member, _sa_initiator=initiator)
844 elif member in constants:
845 appender(member, _sa_initiator=False)
847 if existing_adapter:
848 for member in removals:
849 existing_adapter.fire_remove_event(member, initiator=initiator)
852def prepare_instrumentation(factory):
853 """Prepare a callable for future use as a collection class factory.
855 Given a collection class factory (either a type or no-arg callable),
856 return another factory that will produce compatible instances when
857 called.
859 This function is responsible for converting collection_class=list
860 into the run-time behavior of collection_class=InstrumentedList.
862 """
863 # Convert a builtin to 'Instrumented*'
864 if factory in __canned_instrumentation:
865 factory = __canned_instrumentation[factory]
867 # Create a specimen
868 cls = type(factory())
870 # Did factory callable return a builtin?
871 if cls in __canned_instrumentation:
872 # Wrap it so that it returns our 'Instrumented*'
873 factory = __converting_factory(cls, factory)
874 cls = factory()
876 # Instrument the class if needed.
877 if __instrumentation_mutex.acquire():
878 try:
879 if getattr(cls, "_sa_instrumented", None) != id(cls):
880 _instrument_class(cls)
881 finally:
882 __instrumentation_mutex.release()
884 return factory
887def __converting_factory(specimen_cls, original_factory):
888 """Return a wrapper that converts a "canned" collection like
889 set, dict, list into the Instrumented* version.
891 """
893 instrumented_cls = __canned_instrumentation[specimen_cls]
895 def wrapper():
896 collection = original_factory()
897 return instrumented_cls(collection)
899 # often flawed but better than nothing
900 wrapper.__name__ = "%sWrapper" % original_factory.__name__
901 wrapper.__doc__ = original_factory.__doc__
903 return wrapper
906def _instrument_class(cls):
907 """Modify methods in a class and install instrumentation."""
909 # In the normal call flow, a request for any of the 3 basic collection
910 # types is transformed into one of our trivial subclasses
911 # (e.g. InstrumentedList). Catch anything else that sneaks in here...
912 if cls.__module__ == "__builtin__":
913 raise sa_exc.ArgumentError(
914 "Can not instrument a built-in type. Use a "
915 "subclass, even a trivial one."
916 )
918 roles, methods = _locate_roles_and_methods(cls)
920 _setup_canned_roles(cls, roles, methods)
922 _assert_required_roles(cls, roles, methods)
924 _set_collection_attributes(cls, roles, methods)
927def _locate_roles_and_methods(cls):
928 """search for _sa_instrument_role-decorated methods in
929 method resolution order, assign to roles.
931 """
933 roles = {}
934 methods = {}
936 for supercls in cls.__mro__:
937 for name, method in vars(supercls).items():
938 if not callable(method):
939 continue
941 # note role declarations
942 if hasattr(method, "_sa_instrument_role"):
943 role = method._sa_instrument_role
944 assert role in (
945 "appender",
946 "remover",
947 "iterator",
948 "converter",
949 )
950 roles.setdefault(role, name)
952 # transfer instrumentation requests from decorated function
953 # to the combined queue
954 before, after = None, None
955 if hasattr(method, "_sa_instrument_before"):
956 op, argument = method._sa_instrument_before
957 assert op in ("fire_append_event", "fire_remove_event")
958 before = op, argument
959 if hasattr(method, "_sa_instrument_after"):
960 op = method._sa_instrument_after
961 assert op in ("fire_append_event", "fire_remove_event")
962 after = op
963 if before:
964 methods[name] = before + (after,)
965 elif after:
966 methods[name] = None, None, after
967 return roles, methods
970def _setup_canned_roles(cls, roles, methods):
971 """see if this class has "canned" roles based on a known
972 collection type (dict, set, list). Apply those roles
973 as needed to the "roles" dictionary, and also
974 prepare "decorator" methods
976 """
977 collection_type = util.duck_type_collection(cls)
978 if collection_type in __interfaces:
979 canned_roles, decorators = __interfaces[collection_type]
980 for role, name in canned_roles.items():
981 roles.setdefault(role, name)
983 # apply ABC auto-decoration to methods that need it
984 for method, decorator in decorators.items():
985 fn = getattr(cls, method, None)
986 if (
987 fn
988 and method not in methods
989 and not hasattr(fn, "_sa_instrumented")
990 ):
991 setattr(cls, method, decorator(fn))
994def _assert_required_roles(cls, roles, methods):
995 """ensure all roles are present, and apply implicit instrumentation if
996 needed
998 """
999 if "appender" not in roles or not hasattr(cls, roles["appender"]):
1000 raise sa_exc.ArgumentError(
1001 "Type %s must elect an appender method to be "
1002 "a collection class" % cls.__name__
1003 )
1004 elif roles["appender"] not in methods and not hasattr(
1005 getattr(cls, roles["appender"]), "_sa_instrumented"
1006 ):
1007 methods[roles["appender"]] = ("fire_append_event", 1, None)
1009 if "remover" not in roles or not hasattr(cls, roles["remover"]):
1010 raise sa_exc.ArgumentError(
1011 "Type %s must elect a remover method to be "
1012 "a collection class" % cls.__name__
1013 )
1014 elif roles["remover"] not in methods and not hasattr(
1015 getattr(cls, roles["remover"]), "_sa_instrumented"
1016 ):
1017 methods[roles["remover"]] = ("fire_remove_event", 1, None)
1019 if "iterator" not in roles or not hasattr(cls, roles["iterator"]):
1020 raise sa_exc.ArgumentError(
1021 "Type %s must elect an iterator method to be "
1022 "a collection class" % cls.__name__
1023 )
1026def _set_collection_attributes(cls, roles, methods):
1027 """apply ad-hoc instrumentation from decorators, class-level defaults
1028 and implicit role declarations
1030 """
1031 for method_name, (before, argument, after) in methods.items():
1032 setattr(
1033 cls,
1034 method_name,
1035 _instrument_membership_mutator(
1036 getattr(cls, method_name), before, argument, after
1037 ),
1038 )
1039 # intern the role map
1040 for role, method_name in roles.items():
1041 setattr(cls, "_sa_%s" % role, getattr(cls, method_name))
1043 cls._sa_adapter = None
1045 if not hasattr(cls, "_sa_converter"):
1046 cls._sa_converter = None
1047 cls._sa_instrumented = id(cls)
1050def _instrument_membership_mutator(method, before, argument, after):
1051 """Route method args and/or return value through the collection
1052 adapter."""
1053 # This isn't smart enough to handle @adds(1) for 'def fn(self, (a, b))'
1054 if before:
1055 fn_args = list(
1056 util.flatten_iterator(inspect_getfullargspec(method)[0])
1057 )
1058 if isinstance(argument, int):
1059 pos_arg = argument
1060 named_arg = len(fn_args) > argument and fn_args[argument] or None
1061 else:
1062 if argument in fn_args:
1063 pos_arg = fn_args.index(argument)
1064 else:
1065 pos_arg = None
1066 named_arg = argument
1067 del fn_args
1069 def wrapper(*args, **kw):
1070 if before:
1071 if pos_arg is None:
1072 if named_arg not in kw:
1073 raise sa_exc.ArgumentError(
1074 "Missing argument %s" % argument
1075 )
1076 value = kw[named_arg]
1077 else:
1078 if len(args) > pos_arg:
1079 value = args[pos_arg]
1080 elif named_arg in kw:
1081 value = kw[named_arg]
1082 else:
1083 raise sa_exc.ArgumentError(
1084 "Missing argument %s" % argument
1085 )
1087 initiator = kw.pop("_sa_initiator", None)
1088 if initiator is False:
1089 executor = None
1090 else:
1091 executor = args[0]._sa_adapter
1093 if before and executor:
1094 getattr(executor, before)(value, initiator)
1096 if not after or not executor:
1097 return method(*args, **kw)
1098 else:
1099 res = method(*args, **kw)
1100 if res is not None:
1101 getattr(executor, after)(res, initiator)
1102 return res
1104 wrapper._sa_instrumented = True
1105 if hasattr(method, "_sa_instrument_role"):
1106 wrapper._sa_instrument_role = method._sa_instrument_role
1107 wrapper.__name__ = method.__name__
1108 wrapper.__doc__ = method.__doc__
1109 return wrapper
1112def __set_wo_mutation(collection, item, _sa_initiator=None):
1113 """Run set wo mutation events.
1115 The collection is not mutated.
1117 """
1118 if _sa_initiator is not False:
1119 executor = collection._sa_adapter
1120 if executor:
1121 executor.fire_append_wo_mutation_event(item, _sa_initiator)
1124def __set(collection, item, _sa_initiator=None):
1125 """Run set events.
1127 This event always occurs before the collection is actually mutated.
1129 """
1131 if _sa_initiator is not False:
1132 executor = collection._sa_adapter
1133 if executor:
1134 item = executor.fire_append_event(item, _sa_initiator)
1135 return item
1138def __del(collection, item, _sa_initiator=None):
1139 """Run del events.
1141 This event occurs before the collection is actually mutated, *except*
1142 in the case of a pop operation, in which case it occurs afterwards.
1143 For pop operations, the __before_pop hook is called before the
1144 operation occurs.
1146 """
1147 if _sa_initiator is not False:
1148 executor = collection._sa_adapter
1149 if executor:
1150 executor.fire_remove_event(item, _sa_initiator)
1153def __before_pop(collection, _sa_initiator=None):
1154 """An event which occurs on a before a pop() operation occurs."""
1155 executor = collection._sa_adapter
1156 if executor:
1157 executor.fire_pre_remove_event(_sa_initiator)
1160def _list_decorators():
1161 """Tailored instrumentation wrappers for any list-like class."""
1163 def _tidy(fn):
1164 fn._sa_instrumented = True
1165 fn.__doc__ = getattr(list, fn.__name__).__doc__
1167 def append(fn):
1168 def append(self, item, _sa_initiator=None):
1169 item = __set(self, item, _sa_initiator)
1170 fn(self, item)
1172 _tidy(append)
1173 return append
1175 def remove(fn):
1176 def remove(self, value, _sa_initiator=None):
1177 __del(self, value, _sa_initiator)
1178 # testlib.pragma exempt:__eq__
1179 fn(self, value)
1181 _tidy(remove)
1182 return remove
1184 def insert(fn):
1185 def insert(self, index, value):
1186 value = __set(self, value)
1187 fn(self, index, value)
1189 _tidy(insert)
1190 return insert
1192 def __setitem__(fn):
1193 def __setitem__(self, index, value):
1194 if not isinstance(index, slice):
1195 existing = self[index]
1196 if existing is not None:
1197 __del(self, existing)
1198 value = __set(self, value)
1199 fn(self, index, value)
1200 else:
1201 # slice assignment requires __delitem__, insert, __len__
1202 step = index.step or 1
1203 start = index.start or 0
1204 if start < 0:
1205 start += len(self)
1206 if index.stop is not None:
1207 stop = index.stop
1208 else:
1209 stop = len(self)
1210 if stop < 0:
1211 stop += len(self)
1213 if step == 1:
1214 if value is self:
1215 return
1216 for i in range(start, stop, step):
1217 if len(self) > start:
1218 del self[start]
1220 for i, item in enumerate(value):
1221 self.insert(i + start, item)
1222 else:
1223 rng = list(range(start, stop, step))
1224 if len(value) != len(rng):
1225 raise ValueError(
1226 "attempt to assign sequence of size %s to "
1227 "extended slice of size %s"
1228 % (len(value), len(rng))
1229 )
1230 for i, item in zip(rng, value):
1231 self.__setitem__(i, item)
1233 _tidy(__setitem__)
1234 return __setitem__
1236 def __delitem__(fn):
1237 def __delitem__(self, index):
1238 if not isinstance(index, slice):
1239 item = self[index]
1240 __del(self, item)
1241 fn(self, index)
1242 else:
1243 # slice deletion requires __getslice__ and a slice-groking
1244 # __getitem__ for stepped deletion
1245 # note: not breaking this into atomic dels
1246 for item in self[index]:
1247 __del(self, item)
1248 fn(self, index)
1250 _tidy(__delitem__)
1251 return __delitem__
1253 if util.py2k:
1255 def __setslice__(fn):
1256 def __setslice__(self, start, end, values):
1257 for value in self[start:end]:
1258 __del(self, value)
1259 values = [__set(self, value) for value in values]
1260 fn(self, start, end, values)
1262 _tidy(__setslice__)
1263 return __setslice__
1265 def __delslice__(fn):
1266 def __delslice__(self, start, end):
1267 for value in self[start:end]:
1268 __del(self, value)
1269 fn(self, start, end)
1271 _tidy(__delslice__)
1272 return __delslice__
1274 def extend(fn):
1275 def extend(self, iterable):
1276 for value in list(iterable):
1277 self.append(value)
1279 _tidy(extend)
1280 return extend
1282 def __iadd__(fn):
1283 def __iadd__(self, iterable):
1284 # list.__iadd__ takes any iterable and seems to let TypeError
1285 # raise as-is instead of returning NotImplemented
1286 for value in list(iterable):
1287 self.append(value)
1288 return self
1290 _tidy(__iadd__)
1291 return __iadd__
1293 def pop(fn):
1294 def pop(self, index=-1):
1295 __before_pop(self)
1296 item = fn(self, index)
1297 __del(self, item)
1298 return item
1300 _tidy(pop)
1301 return pop
1303 if not util.py2k:
1305 def clear(fn):
1306 def clear(self, index=-1):
1307 for item in self:
1308 __del(self, item)
1309 fn(self)
1311 _tidy(clear)
1312 return clear
1314 # __imul__ : not wrapping this. all members of the collection are already
1315 # present, so no need to fire appends... wrapping it with an explicit
1316 # decorator is still possible, so events on *= can be had if they're
1317 # desired. hard to imagine a use case for __imul__, though.
1319 l = locals().copy()
1320 l.pop("_tidy")
1321 return l
1324def _dict_decorators():
1325 """Tailored instrumentation wrappers for any dict-like mapping class."""
1327 def _tidy(fn):
1328 fn._sa_instrumented = True
1329 fn.__doc__ = getattr(dict, fn.__name__).__doc__
1331 Unspecified = util.symbol("Unspecified")
1333 def __setitem__(fn):
1334 def __setitem__(self, key, value, _sa_initiator=None):
1335 if key in self:
1336 __del(self, self[key], _sa_initiator)
1337 value = __set(self, value, _sa_initiator)
1338 fn(self, key, value)
1340 _tidy(__setitem__)
1341 return __setitem__
1343 def __delitem__(fn):
1344 def __delitem__(self, key, _sa_initiator=None):
1345 if key in self:
1346 __del(self, self[key], _sa_initiator)
1347 fn(self, key)
1349 _tidy(__delitem__)
1350 return __delitem__
1352 def clear(fn):
1353 def clear(self):
1354 for key in self:
1355 __del(self, self[key])
1356 fn(self)
1358 _tidy(clear)
1359 return clear
1361 def pop(fn):
1362 def pop(self, key, default=Unspecified):
1363 __before_pop(self)
1364 _to_del = key in self
1365 if default is Unspecified:
1366 item = fn(self, key)
1367 else:
1368 item = fn(self, key, default)
1369 if _to_del:
1370 __del(self, item)
1371 return item
1373 _tidy(pop)
1374 return pop
1376 def popitem(fn):
1377 def popitem(self):
1378 __before_pop(self)
1379 item = fn(self)
1380 __del(self, item[1])
1381 return item
1383 _tidy(popitem)
1384 return popitem
1386 def setdefault(fn):
1387 def setdefault(self, key, default=None):
1388 if key not in self:
1389 self.__setitem__(key, default)
1390 return default
1391 else:
1392 value = self.__getitem__(key)
1393 if value is default:
1394 __set_wo_mutation(self, value, None)
1396 return value
1398 _tidy(setdefault)
1399 return setdefault
1401 def update(fn):
1402 def update(self, __other=Unspecified, **kw):
1403 if __other is not Unspecified:
1404 if hasattr(__other, "keys"):
1405 for key in list(__other):
1406 if key not in self or self[key] is not __other[key]:
1407 self[key] = __other[key]
1408 else:
1409 __set_wo_mutation(self, __other[key], None)
1410 else:
1411 for key, value in __other:
1412 if key not in self or self[key] is not value:
1413 self[key] = value
1414 else:
1415 __set_wo_mutation(self, value, None)
1416 for key in kw:
1417 if key not in self or self[key] is not kw[key]:
1418 self[key] = kw[key]
1419 else:
1420 __set_wo_mutation(self, kw[key], None)
1422 _tidy(update)
1423 return update
1425 l = locals().copy()
1426 l.pop("_tidy")
1427 l.pop("Unspecified")
1428 return l
1431_set_binop_bases = (set, frozenset)
1434def _set_binops_check_strict(self, obj):
1435 """Allow only set, frozenset and self.__class__-derived
1436 objects in binops."""
1437 return isinstance(obj, _set_binop_bases + (self.__class__,))
1440def _set_binops_check_loose(self, obj):
1441 """Allow anything set-like to participate in set binops."""
1442 return (
1443 isinstance(obj, _set_binop_bases + (self.__class__,))
1444 or util.duck_type_collection(obj) == set
1445 )
1448def _set_decorators():
1449 """Tailored instrumentation wrappers for any set-like class."""
1451 def _tidy(fn):
1452 fn._sa_instrumented = True
1453 fn.__doc__ = getattr(set, fn.__name__).__doc__
1455 Unspecified = util.symbol("Unspecified")
1457 def add(fn):
1458 def add(self, value, _sa_initiator=None):
1459 if value not in self:
1460 value = __set(self, value, _sa_initiator)
1461 else:
1462 __set_wo_mutation(self, value, _sa_initiator)
1463 # testlib.pragma exempt:__hash__
1464 fn(self, value)
1466 _tidy(add)
1467 return add
1469 def discard(fn):
1470 def discard(self, value, _sa_initiator=None):
1471 # testlib.pragma exempt:__hash__
1472 if value in self:
1473 __del(self, value, _sa_initiator)
1474 # testlib.pragma exempt:__hash__
1475 fn(self, value)
1477 _tidy(discard)
1478 return discard
1480 def remove(fn):
1481 def remove(self, value, _sa_initiator=None):
1482 # testlib.pragma exempt:__hash__
1483 if value in self:
1484 __del(self, value, _sa_initiator)
1485 # testlib.pragma exempt:__hash__
1486 fn(self, value)
1488 _tidy(remove)
1489 return remove
1491 def pop(fn):
1492 def pop(self):
1493 __before_pop(self)
1494 item = fn(self)
1495 # for set in particular, we have no way to access the item
1496 # that will be popped before pop is called.
1497 __del(self, item)
1498 return item
1500 _tidy(pop)
1501 return pop
1503 def clear(fn):
1504 def clear(self):
1505 for item in list(self):
1506 self.remove(item)
1508 _tidy(clear)
1509 return clear
1511 def update(fn):
1512 def update(self, value):
1513 for item in value:
1514 self.add(item)
1516 _tidy(update)
1517 return update
1519 def __ior__(fn):
1520 def __ior__(self, value):
1521 if not _set_binops_check_strict(self, value):
1522 return NotImplemented
1523 for item in value:
1524 self.add(item)
1525 return self
1527 _tidy(__ior__)
1528 return __ior__
1530 def difference_update(fn):
1531 def difference_update(self, value):
1532 for item in value:
1533 self.discard(item)
1535 _tidy(difference_update)
1536 return difference_update
1538 def __isub__(fn):
1539 def __isub__(self, value):
1540 if not _set_binops_check_strict(self, value):
1541 return NotImplemented
1542 for item in value:
1543 self.discard(item)
1544 return self
1546 _tidy(__isub__)
1547 return __isub__
1549 def intersection_update(fn):
1550 def intersection_update(self, other):
1551 want, have = self.intersection(other), set(self)
1552 remove, add = have - want, want - have
1554 for item in remove:
1555 self.remove(item)
1556 for item in add:
1557 self.add(item)
1559 _tidy(intersection_update)
1560 return intersection_update
1562 def __iand__(fn):
1563 def __iand__(self, other):
1564 if not _set_binops_check_strict(self, other):
1565 return NotImplemented
1566 want, have = self.intersection(other), set(self)
1567 remove, add = have - want, want - have
1569 for item in remove:
1570 self.remove(item)
1571 for item in add:
1572 self.add(item)
1573 return self
1575 _tidy(__iand__)
1576 return __iand__
1578 def symmetric_difference_update(fn):
1579 def symmetric_difference_update(self, other):
1580 want, have = self.symmetric_difference(other), set(self)
1581 remove, add = have - want, want - have
1583 for item in remove:
1584 self.remove(item)
1585 for item in add:
1586 self.add(item)
1588 _tidy(symmetric_difference_update)
1589 return symmetric_difference_update
1591 def __ixor__(fn):
1592 def __ixor__(self, other):
1593 if not _set_binops_check_strict(self, other):
1594 return NotImplemented
1595 want, have = self.symmetric_difference(other), set(self)
1596 remove, add = have - want, want - have
1598 for item in remove:
1599 self.remove(item)
1600 for item in add:
1601 self.add(item)
1602 return self
1604 _tidy(__ixor__)
1605 return __ixor__
1607 l = locals().copy()
1608 l.pop("_tidy")
1609 l.pop("Unspecified")
1610 return l
1613class InstrumentedList(list):
1614 """An instrumented version of the built-in list."""
1617class InstrumentedSet(set):
1618 """An instrumented version of the built-in set."""
1621class InstrumentedDict(dict):
1622 """An instrumented version of the built-in dict."""
1625__canned_instrumentation = {
1626 list: InstrumentedList,
1627 set: InstrumentedSet,
1628 dict: InstrumentedDict,
1629}
1631__interfaces = {
1632 list: (
1633 {"appender": "append", "remover": "remove", "iterator": "__iter__"},
1634 _list_decorators(),
1635 ),
1636 set: (
1637 {"appender": "add", "remover": "remove", "iterator": "__iter__"},
1638 _set_decorators(),
1639 ),
1640 # decorators are required for dicts and object collections.
1641 dict: ({"iterator": "values"}, _dict_decorators())
1642 if util.py3k
1643 else ({"iterator": "itervalues"}, _dict_decorators()),
1644}
1647class MappedCollection(dict):
1648 """A basic dictionary-based collection class.
1650 Extends dict with the minimal bag semantics that collection
1651 classes require. ``set`` and ``remove`` are implemented in terms
1652 of a keying function: any callable that takes an object and
1653 returns an object for use as a dictionary key.
1655 """
1657 def __init__(self, keyfunc):
1658 """Create a new collection with keying provided by keyfunc.
1660 keyfunc may be any callable that takes an object and returns an object
1661 for use as a dictionary key.
1663 The keyfunc will be called every time the ORM needs to add a member by
1664 value-only (such as when loading instances from the database) or
1665 remove a member. The usual cautions about dictionary keying apply-
1666 ``keyfunc(object)`` should return the same output for the life of the
1667 collection. Keying based on mutable properties can result in
1668 unreachable instances "lost" in the collection.
1670 """
1671 self.keyfunc = keyfunc
1673 @collection.appender
1674 @collection.internally_instrumented
1675 def set(self, value, _sa_initiator=None):
1676 """Add an item by value, consulting the keyfunc for the key."""
1678 key = self.keyfunc(value)
1679 self.__setitem__(key, value, _sa_initiator)
1681 @collection.remover
1682 @collection.internally_instrumented
1683 def remove(self, value, _sa_initiator=None):
1684 """Remove an item by value, consulting the keyfunc for the key."""
1686 key = self.keyfunc(value)
1687 # Let self[key] raise if key is not in this collection
1688 # testlib.pragma exempt:__ne__
1689 if self[key] != value:
1690 raise sa_exc.InvalidRequestError(
1691 "Can not remove '%s': collection holds '%s' for key '%s'. "
1692 "Possible cause: is the MappedCollection key function "
1693 "based on mutable properties or properties that only obtain "
1694 "values after flush?" % (value, self[key], key)
1695 )
1696 self.__delitem__(key, _sa_initiator)
1699# ensure instrumentation is associated with
1700# these built-in classes; if a user-defined class
1701# subclasses these and uses @internally_instrumented,
1702# the superclass is otherwise not instrumented.
1703# see [ticket:2406].
1704_instrument_class(MappedCollection)
1705_instrument_class(InstrumentedList)
1706_instrument_class(InstrumentedSet)