Coverage for /pythoncovmergedfiles/medio/medio/usr/local/lib/python3.11/site-packages/dulwich/line_ending.py: 34%
Shortcuts on this page
r m x toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
1# line_ending.py -- Line ending conversion functions
2# Copyright (C) 2018-2018 Boris Feld <boris.feld@comet.ml>
3#
4# SPDX-License-Identifier: Apache-2.0 OR GPL-2.0-or-later
5# Dulwich is dual-licensed under the Apache License, Version 2.0 and the GNU
6# General Public License as published by the Free Software Foundation; version 2.0
7# or (at your option) any later version. You can redistribute it and/or
8# modify it under the terms of either of these two licenses.
9#
10# Unless required by applicable law or agreed to in writing, software
11# distributed under the License is distributed on an "AS IS" BASIS,
12# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13# See the License for the specific language governing permissions and
14# limitations under the License.
15#
16# You should have received a copy of the licenses; if not, see
17# <http://www.gnu.org/licenses/> for a copy of the GNU General Public License
18# and <http://www.apache.org/licenses/LICENSE-2.0> for a copy of the Apache
19# License, Version 2.0.
20#
21r"""All line-ending related functions, from conversions to config processing.
23Line-ending normalization is a complex beast. Here is some notes and details
24about how it seems to work.
26The normalization is a two-fold process that happens at two moments:
28- When reading a file from the index and to the working directory. For example
29 when doing a ``git clone`` or ``git checkout`` call. This is called the
30 smudge filter (repository -> working tree).
31- When writing a file to the index from the working directory. For example
32 when doing a ``git add`` call. This is called the clean filter (working tree
33 -> repository).
35Note that when checking status (getting unstaged changes), whether or not
36normalization is done on write depends on whether or not the file in the
37working dir has also been normalized on read:
39- For autocrlf=true all files are always normalized on both read and write.
40- For autocrlf=input files are only normalized on write if they are newly
41 "added". Since files which are already committed are not normalized on
42 checkout into the working tree, they are also left alone when staging
43 modifications into the index.
45One thing to know is that Git does line-ending normalization only on text
46files. How does Git know that a file is text? We can either mark a file as a
47text file, a binary file or ask Git to automatically decides. Git has an
48heuristic to detect if a file is a text file or a binary file. It seems based
49on the percentage of non-printable characters in files.
51The code for this heuristic is here:
52https://git.kernel.org/pub/scm/git/git.git/tree/convert.c#n46
54Dulwich have an implementation with a slightly different heuristic, the
55`dulwich.patch.is_binary` function.
57The binary detection heuristic implementation is close to the one in JGit:
58https://github.com/eclipse/jgit/blob/f6873ffe522bbc3536969a3a3546bf9a819b92bf/org.eclipse.jgit/src/org/eclipse/jgit/diff/RawText.java#L300
60There is multiple variables that impact the normalization.
62First, a repository can contains a ``.gitattributes`` file (or more than one...)
63that can further customize the operation on some file patterns, for example:
65 \*.txt text
67Force all ``.txt`` files to be treated as text files and to have their lines
68endings normalized.
70 \*.jpg -text
72Force all ``.jpg`` files to be treated as binary files and to not have their
73lines endings converted.
75 \*.vcproj text eol=crlf
77Force all ``.vcproj`` files to be treated as text files and to have their lines
78endings converted into ``CRLF`` in working directory no matter the native EOL of
79the platform.
81 \*.sh text eol=lf
83Force all ``.sh`` files to be treated as text files and to have their lines
84endings converted into ``LF`` in working directory no matter the native EOL of
85the platform.
87If the ``eol`` attribute is not defined, Git uses the ``core.eol`` configuration
88value described later.
90 \* text=auto
92Force all files to be scanned by the text file heuristic detection and to have
93their line endings normalized in case they are detected as text files.
95Git also have a obsolete attribute named ``crlf`` that can be translated to the
96corresponding text attribute value.
98Then there are some configuration option (that can be defined at the
99repository or user level):
101- core.autocrlf
102- core.eol
104``core.autocrlf`` is taken into account for all files that doesn't have a ``text``
105attribute defined in ``.gitattributes``; it takes three possible values:
107 - ``true``: This forces all files on the working directory to have CRLF
108 line-endings in the working directory and convert line-endings to LF
109 when writing to the index. An explicit ``eol`` attribute still takes
110 precedence over it.
111 - ``input``: Quite similar to the ``true`` value but only applies the clean
112 filter, ie line-ending of new files added to the index will get their
113 line-endings converted to LF.
114 - ``false`` (default): No normalization is done.
116``core.eol`` is the top-level configuration to define the line-ending to use
117when applying the smudge filter. It takes three possible values:
119 - ``lf``: When normalization is done, force line-endings to be ``LF`` in the
120 working directory.
121 - ``crlf``: When normalization is done, force line-endings to be ``CRLF`` in
122 the working directory.
123 - ``native`` (default): When normalization is done, force line-endings to be
124 the platform's native line ending.
126One thing to remember is when line-ending normalization is done on a file, Git
127always normalize line-ending to ``LF`` when writing to the index.
129There are sources that seems to indicate that Git won't do line-ending
130normalization when a file contains mixed line-endings. I think this logic
131might be in text / binary detection heuristic but couldn't find it yet.
133Sources:
134- https://git-scm.com/docs/git-config#git-config-coreeol
135- https://git-scm.com/docs/git-config#git-config-coreautocrlf
136- https://git-scm.com/docs/gitattributes#_checking_out_and_checking_in
137- https://adaptivepatchwork.com/2012/03/01/mind-the-end-of-your-line/
138"""
140__all__ = [
141 "CRLF",
142 "LF",
143 "BlobNormalizer",
144 "CRLFAction",
145 "LineEndingFilter",
146 "TreeBlobNormalizer",
147 "check_safecrlf",
148 "convert_crlf_to_lf",
149 "convert_lf_to_crlf",
150 "get_clean_filter",
151 "get_clean_filter_autocrlf",
152 "get_smudge_filter",
153 "get_smudge_filter_autocrlf",
154 "line_ending_filter_for_action",
155 "normalize_blob",
156 "read_eol_config",
157 "resolve_crlf_action",
158]
160import logging
161import os
162from collections.abc import Callable, Mapping
163from enum import Enum
164from typing import TYPE_CHECKING, Any
166if TYPE_CHECKING:
167 from .config import Config
168 from .object_store import BaseObjectStore
170from .attrs import AttributeValue, GitAttributes, Pattern
171from .filters import FilterBlobNormalizer, FilterContext, FilterRegistry
172from .object_store import iter_tree_contents
173from .objects import Blob, ObjectID
174from .patch import is_binary
175from .wildmatch import MalformedPattern
177CRLF = b"\r\n"
178LF = b"\n"
180logger = logging.getLogger(__name__)
183def read_eol_config(config: "Config | None") -> tuple[str, bytes, bytes]:
184 """Read the line ending related settings from a configuration.
186 Returns: Tuple with the values of ``core.eol``, ``core.autocrlf`` and
187 ``core.safecrlf``
188 """
189 if config is None:
190 return "native", b"false", b"false"
192 try:
193 core_eol_raw = config.get("core", "eol")
194 except KeyError:
195 core_eol = "native"
196 else:
197 core_eol = (
198 core_eol_raw.decode("ascii")
199 if isinstance(core_eol_raw, bytes)
200 else str(core_eol_raw)
201 ).lower()
203 try:
204 autocrlf_raw = config.get("core", "autocrlf")
205 except KeyError:
206 autocrlf = b"false"
207 else:
208 autocrlf = (
209 autocrlf_raw.lower()
210 if isinstance(autocrlf_raw, bytes)
211 else str(autocrlf_raw).lower().encode("ascii")
212 )
214 try:
215 safecrlf_raw = config.get("core", "safecrlf")
216 except KeyError:
217 safecrlf = b"false"
218 else:
219 safecrlf = (
220 safecrlf_raw
221 if isinstance(safecrlf_raw, bytes)
222 else safecrlf_raw.encode("utf-8")
223 )
225 return core_eol, autocrlf, safecrlf
228class CRLFAction(Enum):
229 """The line ending conversion to apply to a path.
231 This mirrors git's ``enum convert_crlf_action`` in convert.c.
232 ``UNDEFINED``, ``TEXT`` and ``AUTO`` are intermediate states only;
233 :func:`resolve_crlf_action` always resolves them into one of the
234 others.
235 """
237 UNDEFINED = "undefined"
238 BINARY = "binary"
239 TEXT = "text"
240 TEXT_INPUT = "text-input"
241 TEXT_CRLF = "text-crlf"
242 AUTO = "auto"
243 AUTO_INPUT = "auto-input"
244 AUTO_CRLF = "auto-crlf"
247# Actions where git applies its text/binary heuristic rather than trusting the
248# attributes, and so refuses to touch files that already contain CR.
249_AUTO_ACTIONS = (CRLFAction.AUTO, CRLFAction.AUTO_INPUT, CRLFAction.AUTO_CRLF)
252def _crlf_action_from_attr(value: AttributeValue) -> CRLFAction:
253 """Translate a ``text`` (or obsolete ``crlf``) attribute into an action."""
254 match value:
255 case True:
256 return CRLFAction.TEXT
257 case False:
258 return CRLFAction.BINARY
259 case b"input":
260 return CRLFAction.TEXT_INPUT
261 case b"auto":
262 return CRLFAction.AUTO
263 return CRLFAction.UNDEFINED
266def _text_eol_is_crlf(core_eol: str, autocrlf: bytes) -> bool:
267 """Decide the working tree line ending when only the config says so."""
268 if autocrlf == b"true":
269 return True
270 elif autocrlf == b"input":
271 return False
272 if core_eol == "crlf":
273 return True
274 if core_eol == "native":
275 return os.linesep == "\r\n"
276 return False
279def resolve_crlf_action(
280 attributes: Mapping[bytes, AttributeValue],
281 core_eol: str = "native",
282 autocrlf: bytes = b"false",
283) -> CRLFAction:
284 """Determine the line ending conversion for a path.
286 This follows ``convert_attrs`` in git's convert.c: the ``text`` attribute
287 (or the obsolete ``crlf`` one) selects whether conversion happens at all,
288 an explicit ``eol`` attribute then pins the working tree line ending, and
289 ``core.autocrlf``/``core.eol`` only apply where the attributes are silent.
291 Args:
292 attributes: Attributes matched for the path
293 core_eol: Value of ``core.eol``
294 autocrlf: Value of ``core.autocrlf``
296 Returns: The action to apply
297 """
298 action = _crlf_action_from_attr(attributes.get(b"text"))
299 if action == CRLFAction.UNDEFINED:
300 action = _crlf_action_from_attr(attributes.get(b"crlf"))
302 if action != CRLFAction.BINARY:
303 eol_attr = attributes.get(b"eol")
304 if action == CRLFAction.AUTO and eol_attr == b"lf":
305 action = CRLFAction.AUTO_INPUT
306 elif action == CRLFAction.AUTO and eol_attr == b"crlf":
307 action = CRLFAction.AUTO_CRLF
308 elif eol_attr == b"lf":
309 action = CRLFAction.TEXT_INPUT
310 elif eol_attr == b"crlf":
311 action = CRLFAction.TEXT_CRLF
313 if action in (CRLFAction.TEXT, CRLFAction.AUTO):
314 # Neither the attributes nor an "eol" attribute pinned the working
315 # tree line ending, so fall back to the configuration.
316 crlf = _text_eol_is_crlf(core_eol, autocrlf)
317 if action == CRLFAction.TEXT:
318 action = CRLFAction.TEXT_CRLF if crlf else CRLFAction.TEXT_INPUT
319 else:
320 action = CRLFAction.AUTO_CRLF if crlf else CRLFAction.AUTO_INPUT
321 if action == CRLFAction.UNDEFINED:
322 if autocrlf == b"true":
323 action = CRLFAction.AUTO_CRLF
324 elif autocrlf == b"input":
325 action = CRLFAction.AUTO_INPUT
326 else:
327 action = CRLFAction.BINARY
329 return action
332def line_ending_filter_for_action(
333 action: CRLFAction,
334 safecrlf: bytes = b"false",
335 index_has_crlf: Callable[[], bool] | None = None,
336) -> "LineEndingFilter | None":
337 """Build the filter implementing a resolved :class:`CRLFAction`.
339 Args:
340 action: Action as returned by :func:`resolve_crlf_action`
341 safecrlf: Value of ``core.safecrlf``
342 index_has_crlf: Callback reporting whether the index already holds a
343 CRLF text blob for the path; only consulted for the "auto" actions
345 Returns: A filter, or None if the action converts nothing
346 """
347 if action == CRLFAction.BINARY:
348 return None
350 smudge: Callable[[bytes], bytes] | None = (
351 convert_lf_to_crlf
352 if action in (CRLFAction.TEXT_CRLF, CRLFAction.AUTO_CRLF)
353 else None
354 )
355 return LineEndingFilter(
356 clean_conversion=convert_crlf_to_lf,
357 smudge_conversion=smudge,
358 binary_detection=True,
359 safecrlf=safecrlf,
360 # Only the "auto" family leaves files with existing CRs alone; an
361 # explicit "text" attribute means the user has already decided.
362 require_lf_only=action in _AUTO_ACTIONS,
363 index_has_crlf=index_has_crlf if action in _AUTO_ACTIONS else None,
364 )
367class LineEndingFilter:
368 """Filter driver for line ending conversion.
370 Satisfies the :class:`dulwich.filters.FilterDriver` protocol structurally,
371 without inheriting from it, so that the line ending logic this module
372 exports can be imported from ``filters`` without a circular import.
373 """
375 def __init__(
376 self,
377 clean_conversion: Callable[[bytes], bytes] | None = None,
378 smudge_conversion: Callable[[bytes], bytes] | None = None,
379 binary_detection: bool = True,
380 safecrlf: bytes = b"false",
381 require_lf_only: bool = False,
382 index_has_crlf: Callable[[], bool] | None = None,
383 ):
384 """Initialize LineEndingFilter.
386 Args:
387 clean_conversion: Conversion to apply on checkin
388 smudge_conversion: Conversion to apply on checkout
389 binary_detection: Skip files detected as binary
390 safecrlf: Value of ``core.safecrlf``
391 require_lf_only: Only smudge content whose line endings are all LF,
392 as git does for ``text=auto`` and ``core.autocrlf``
393 index_has_crlf: Callback reporting whether the index already holds
394 a CRLF text blob for the path, in which case clean leaves CRLF
395 alone, as git does for ``text=auto`` and ``core.autocrlf``
396 """
397 self.clean_conversion = clean_conversion
398 self.smudge_conversion = smudge_conversion
399 self.binary_detection = binary_detection
400 self.safecrlf = safecrlf
401 self.require_lf_only = require_lf_only
402 self.index_has_crlf = index_has_crlf
404 @classmethod
405 def from_config(
406 cls, config: "Config | None", for_text_attr: bool = False
407 ) -> "LineEndingFilter":
408 """Create a LineEndingFilter from git configuration.
410 Args:
411 config: Git configuration
412 for_text_attr: If True, always normalize on checkin (for text attribute)
414 Returns:
415 Configured LineEndingFilter instance
416 """
417 if config is None:
418 # Default filter
419 if for_text_attr:
420 # For text attribute: always normalize on checkin
421 return cls(
422 clean_conversion=convert_crlf_to_lf,
423 smudge_conversion=None,
424 binary_detection=True,
425 )
426 else:
427 # No config: no conversion
428 return cls()
430 core_eol, autocrlf, safecrlf = read_eol_config(config)
432 if for_text_attr:
433 # For text attribute: always normalize to LF on checkin
434 # Smudge behavior depends on core.eol and core.autocrlf
435 smudge_filter = get_smudge_filter(core_eol, autocrlf)
436 clean_filter: Callable[[bytes], bytes] | None = convert_crlf_to_lf
437 else:
438 # Normal autocrlf behavior
439 smudge_filter = get_smudge_filter(core_eol, autocrlf)
440 clean_filter = get_clean_filter(core_eol, autocrlf)
442 return cls(
443 clean_conversion=clean_filter,
444 smudge_conversion=smudge_filter,
445 binary_detection=True,
446 safecrlf=safecrlf,
447 )
449 def clean(self, data: bytes, path: bytes = b"") -> bytes:
450 """Apply line ending conversion for checkin (working tree -> repository)."""
451 if self.clean_conversion is None:
452 return data
454 # Skip binary files if detection is enabled
455 if self.binary_detection and is_binary(data):
456 return data
458 # Keep files that were committed with CRLF stable rather than
459 # rewriting every line on the next edit.
460 if self.index_has_crlf is not None and CRLF in data and self.index_has_crlf():
461 return data
463 converted = self.clean_conversion(data)
465 # Check if conversion is safe
466 if self.safecrlf != b"false":
467 check_safecrlf(data, converted, self.safecrlf, path)
469 return converted
471 def smudge(self, data: bytes, path: bytes = b"") -> bytes:
472 """Apply line ending conversion for checkout (repository -> working tree)."""
473 if self.smudge_conversion is None:
474 return data
476 # Skip binary files if detection is enabled
477 if self.binary_detection and is_binary(data):
478 return data
480 # Git leaves content that already contains CR alone when the decision
481 # came from a heuristic rather than an explicit attribute.
482 if self.require_lf_only and b"\r" in data:
483 return data
485 converted = self.smudge_conversion(data)
487 # Check if conversion is safe
488 if self.safecrlf != b"false":
489 check_safecrlf(data, converted, self.safecrlf, path)
491 return converted
493 def cleanup(self) -> None:
494 """Clean up any resources held by this filter driver."""
495 # LineEndingFilter doesn't hold any resources that need cleanup
497 def reuse(self, config: "Config", filter_name: str) -> bool:
498 """Check if this filter driver should be reused with the given configuration."""
499 # LineEndingFilter is lightweight and should always be recreated
500 # to ensure it uses the latest configuration
501 return False
504def convert_crlf_to_lf(text_hunk: bytes) -> bytes:
505 """Convert CRLF in text hunk into LF.
507 Args:
508 text_hunk: A bytes string representing a text hunk
509 Returns: The text hunk with the same type, with CRLF replaced into LF
510 """
511 return text_hunk.replace(CRLF, LF)
514def convert_lf_to_crlf(text_hunk: bytes) -> bytes:
515 """Convert LF in text hunk into CRLF.
517 Args:
518 text_hunk: A bytes string representing a text hunk
519 Returns: The text hunk with the same type, with LF replaced into CRLF
520 """
521 # Single-pass conversion: split on LF and join with CRLF
522 # This avoids the double replacement issue
523 parts = text_hunk.split(LF)
524 # Remove any trailing CR to avoid CRCRLF
525 cleaned_parts = []
526 for i, part in enumerate(parts):
527 if i < len(parts) - 1 and part.endswith(b"\r"):
528 cleaned_parts.append(part[:-1])
529 else:
530 cleaned_parts.append(part)
531 return CRLF.join(cleaned_parts)
534def check_safecrlf(
535 original: bytes, converted: bytes, safecrlf: bytes, path: bytes = b""
536) -> None:
537 """Check if CRLF conversion is safe according to core.safecrlf setting.
539 Args:
540 original: Original content before conversion
541 converted: Content after conversion
542 safecrlf: Value of core.safecrlf config (b"true", b"warn", or b"false")
543 path: Path to the file being checked (for error messages)
545 Raises:
546 ValueError: If safecrlf is "true" and conversion would lose data
547 """
548 if safecrlf == b"false":
549 return
551 # Check if conversion is reversible
552 if safecrlf in (b"true", b"warn"):
553 # For CRLF->LF conversion, check if converting back would recover original
554 if CRLF in original and CRLF not in converted:
555 # This was a CRLF->LF conversion
556 recovered = convert_lf_to_crlf(converted)
557 if recovered != original:
558 msg = (
559 f"CRLF would be replaced by LF in {path.decode('utf-8', 'replace')}"
560 )
561 if safecrlf == b"true":
562 raise ValueError(msg)
563 else: # warn
564 logger.warning(msg)
566 # For LF->CRLF conversion, check if converting back would recover original
567 elif LF in original and CRLF in converted and CRLF not in original:
568 # This was a LF->CRLF conversion
569 recovered = convert_crlf_to_lf(converted)
570 if recovered != original:
571 msg = (
572 f"LF would be replaced by CRLF in {path.decode('utf-8', 'replace')}"
573 )
574 if safecrlf == b"true":
575 raise ValueError(msg)
576 else: # warn
577 logger.warning(msg)
580def get_smudge_filter(
581 core_eol: str, core_autocrlf: bytes
582) -> Callable[[bytes], bytes] | None:
583 """Returns the correct smudge filter based on the passed arguments."""
584 # Git attributes handling is done by the filter infrastructure
585 return get_smudge_filter_autocrlf(core_autocrlf)
588def get_clean_filter(
589 core_eol: str, core_autocrlf: bytes
590) -> Callable[[bytes], bytes] | None:
591 """Returns the correct clean filter based on the passed arguments."""
592 # Git attributes handling is done by the filter infrastructure
593 return get_clean_filter_autocrlf(core_autocrlf)
596def get_smudge_filter_autocrlf(
597 core_autocrlf: bytes,
598) -> Callable[[bytes], bytes] | None:
599 """Returns the correct smudge filter base on autocrlf value.
601 Args:
602 core_autocrlf: The bytes configuration value of core.autocrlf.
603 Valid values are: b'true', b'false' or b'input'.
604 Returns: Either None if no filter has to be applied or a function
605 accepting a single argument, a binary text hunk
606 """
607 if core_autocrlf == b"true":
608 return convert_lf_to_crlf
610 return None
613def get_clean_filter_autocrlf(
614 core_autocrlf: bytes,
615) -> Callable[[bytes], bytes] | None:
616 """Returns the correct clean filter base on autocrlf value.
618 Args:
619 core_autocrlf: The bytes configuration value of core.autocrlf.
620 Valid values are: b'true', b'false' or b'input'.
621 Returns: Either None if no filter has to be applied or a function
622 accepting a single argument, a binary text hunk
623 """
624 if core_autocrlf == b"true" or core_autocrlf == b"input":
625 return convert_crlf_to_lf
627 # Checking filter should never be `convert_lf_to_crlf`
628 return None
631class BlobNormalizer(FilterBlobNormalizer):
632 """An object to store computation result of which filter to apply based on configuration, gitattributes, path and operation (checkin or checkout).
634 This class maintains backward compatibility while using the filter infrastructure.
635 """
637 def __init__(
638 self,
639 config_stack: "Config",
640 gitattributes: Mapping[str, Any],
641 core_eol: str = "native",
642 autocrlf: bytes = b"false",
643 safecrlf: bytes = b"false",
644 ) -> None:
645 """Initialize FilteringBlobNormalizer."""
646 # Set up a filter registry with line ending filters
647 filter_registry = FilterRegistry(config_stack)
649 # Create line ending filter if needed
650 smudge_filter = get_smudge_filter(core_eol, autocrlf)
651 clean_filter = get_clean_filter(core_eol, autocrlf)
653 # Always register a text filter that can be used by gitattributes
654 # Even if autocrlf is false, gitattributes text=true should work
655 line_ending_filter = LineEndingFilter(
656 clean_conversion=clean_filter or convert_crlf_to_lf,
657 smudge_conversion=smudge_filter or convert_lf_to_crlf,
658 binary_detection=True,
659 safecrlf=safecrlf,
660 )
661 filter_registry.register_driver("text", line_ending_filter)
663 # Convert dict gitattributes to GitAttributes object for parent class
664 git_attrs_patterns = []
665 for pattern_str, attrs in gitattributes.items():
666 if isinstance(pattern_str, str):
667 pattern_bytes = pattern_str.encode("utf-8")
668 else:
669 pattern_bytes = pattern_str
670 try:
671 pattern = Pattern(pattern_bytes)
672 except MalformedPattern:
673 logger.warning("Ignoring malformed pattern %r", pattern_bytes)
674 continue
675 git_attrs_patterns.append((pattern, attrs))
677 git_attributes = GitAttributes(git_attrs_patterns)
679 # Create FilterContext for parent class
680 filter_context = FilterContext(filter_registry)
682 # Initialize parent class with gitattributes
683 # The filter infrastructure will handle gitattributes processing
684 super().__init__(config_stack, git_attributes, filter_context=filter_context)
686 # Store original filters for backward compatibility
687 self.fallback_read_filter = smudge_filter
688 self.fallback_write_filter = clean_filter
690 def checkin_normalize(self, blob: Blob, path: bytes) -> Blob:
691 """Normalize a blob during a checkin operation."""
692 # First try to get filter from gitattributes (handled by parent)
693 result = super().checkin_normalize(blob, path)
695 # Check if gitattributes explicitly disabled text conversion
696 attrs = self.gitattributes.match_path(path)
697 if b"text" in attrs and attrs[b"text"] is False:
698 # Explicitly marked as binary, no conversion
699 return blob
701 # If no filter was applied via gitattributes and we have a fallback filter
702 # (autocrlf is enabled), apply it to all files
703 if result is blob and self.fallback_write_filter is not None:
704 # Apply the clean filter with binary detection
705 # Get safecrlf from config
706 safecrlf = b"false"
707 config_stack = getattr(self.filter_registry, "config_stack", None)
708 if config_stack is not None:
709 safecrlf = config_stack.get(b"core", b"safecrlf", b"false")
710 if hasattr(safecrlf, "encode"):
711 safecrlf = safecrlf.encode("utf-8")
713 line_ending_filter = LineEndingFilter(
714 clean_conversion=self.fallback_write_filter,
715 smudge_conversion=None,
716 binary_detection=True,
717 safecrlf=safecrlf,
718 )
719 filtered_data = line_ending_filter.clean(blob.data, path)
720 if filtered_data != blob.data:
721 new_blob = Blob()
722 new_blob.data = filtered_data
723 return new_blob
725 return result
727 def checkout_normalize(self, blob: Blob, path: bytes) -> Blob:
728 """Normalize a blob during a checkout operation."""
729 # First try to get filter from gitattributes (handled by parent)
730 result = super().checkout_normalize(blob, path)
732 # Check if gitattributes explicitly disabled text conversion
733 attrs = self.gitattributes.match_path(path)
734 if b"text" in attrs and attrs[b"text"] is False:
735 # Explicitly marked as binary, no conversion
736 return blob
738 # If no filter was applied via gitattributes and we have a fallback filter
739 # (autocrlf is enabled), apply it to all files
740 if result is blob and self.fallback_read_filter is not None:
741 # Apply the smudge filter with binary detection
742 # Get safecrlf from config
743 safecrlf = b"false"
744 config_stack = getattr(self.filter_registry, "config_stack", None)
745 if config_stack is not None:
746 safecrlf = config_stack.get(b"core", b"safecrlf", b"false")
747 if hasattr(safecrlf, "encode"):
748 safecrlf = safecrlf.encode("utf-8")
750 line_ending_filter = LineEndingFilter(
751 clean_conversion=None,
752 smudge_conversion=self.fallback_read_filter,
753 binary_detection=True,
754 safecrlf=safecrlf,
755 )
756 filtered_data = line_ending_filter.smudge(blob.data, path)
757 if filtered_data != blob.data:
758 new_blob = Blob()
759 new_blob.data = filtered_data
760 return new_blob
762 return result
765def normalize_blob(
766 blob: Blob, conversion: Callable[[bytes], bytes], binary_detection: bool
767) -> Blob:
768 """Normalize blob by applying line ending conversion."""
769 # Read the original blob
770 data = blob.data
772 # If we need to detect if a file is binary and the file is detected as
773 # binary, do not apply the conversion function and return the original
774 # chunked text
775 if binary_detection is True:
776 if is_binary(data):
777 return blob
779 # Now apply the conversion
780 converted_data = conversion(data)
782 new_blob = Blob()
783 new_blob.data = converted_data
785 return new_blob
788class TreeBlobNormalizer(BlobNormalizer):
789 """Blob normalizer that tracks existing files in a tree."""
791 def __init__(
792 self,
793 config_stack: "Config",
794 git_attributes: Mapping[str, Any],
795 object_store: "BaseObjectStore",
796 tree: ObjectID | None = None,
797 core_eol: str = "native",
798 autocrlf: bytes = b"false",
799 safecrlf: bytes = b"false",
800 ) -> None:
801 """Initialize TreeBlobNormalizer."""
802 super().__init__(config_stack, git_attributes, core_eol, autocrlf, safecrlf)
803 if tree:
804 self.existing_paths = {
805 name for name, _, _ in iter_tree_contents(object_store, tree)
806 }
807 else:
808 self.existing_paths = set()
810 def checkin_normalize(self, blob: Blob, path: bytes) -> Blob:
811 """Normalize blob for checkin, considering existing tree state."""
812 # Existing files should only be normalized on checkin if:
813 # 1. They were previously normalized on checkout (autocrlf=true), OR
814 # 2. We have a write filter (autocrlf=true or autocrlf=input), OR
815 # 3. They are new files
816 if (
817 self.fallback_read_filter is not None
818 or self.fallback_write_filter is not None
819 or path not in self.existing_paths
820 ):
821 return super().checkin_normalize(blob, path)
822 return blob