Coverage for /pythoncovmergedfiles/medio/medio/usr/local/lib/python3.11/site-packages/dulwich/line_ending.py: 35%

1# line_ending.py -- Line ending conversion functions 

2# Copyright (C) 2018-2018 Boris Feld <boris.feld@comet.ml> 

3# 

4# SPDX-License-Identifier: Apache-2.0 OR GPL-2.0-or-later 

5# Dulwich is dual-licensed under the Apache License, Version 2.0 and the GNU 

6# General Public License as published by the Free Software Foundation; version 2.0 

7# or (at your option) any later version. You can redistribute it and/or 

8# modify it under the terms of either of these two licenses. 

9# 

10# Unless required by applicable law or agreed to in writing, software 

11# distributed under the License is distributed on an "AS IS" BASIS, 

12# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 

13# See the License for the specific language governing permissions and 

14# limitations under the License. 

15# 

16# You should have received a copy of the licenses; if not, see 

17# <http://www.gnu.org/licenses/> for a copy of the GNU General Public License 

18# and <http://www.apache.org/licenses/LICENSE-2.0> for a copy of the Apache 

19# License, Version 2.0. 

20# 

21r"""All line-ending related functions, from conversions to config processing. 

22 

23Line-ending normalization is a complex beast. Here is some notes and details 

24about how it seems to work. 

25 

26The normalization is a two-fold process that happens at two moments: 

27 

28- When reading a file from the index and to the working directory. For example 

29 when doing a ``git clone`` or ``git checkout`` call. This is called the 

30 smudge filter (repository -> working tree). 

31- When writing a file to the index from the working directory. For example 

32 when doing a ``git add`` call. This is called the clean filter (working tree 

33 -> repository). 

34 

35Note that when checking status (getting unstaged changes), whether or not 

36normalization is done on write depends on whether or not the file in the 

37working dir has also been normalized on read: 

38 

39- For autocrlf=true all files are always normalized on both read and write. 

40- For autocrlf=input files are only normalized on write if they are newly 

41 "added". Since files which are already committed are not normalized on 

42 checkout into the working tree, they are also left alone when staging 

43 modifications into the index. 

44 

45One thing to know is that Git does line-ending normalization only on text 

46files. How does Git know that a file is text? We can either mark a file as a 

47text file, a binary file or ask Git to automatically decides. Git has an 

48heuristic to detect if a file is a text file or a binary file. It seems based 

49on the percentage of non-printable characters in files. 

50 

51The code for this heuristic is here: 

52https://git.kernel.org/pub/scm/git/git.git/tree/convert.c#n46 

53 

54Dulwich have an implementation with a slightly different heuristic, the 

55`dulwich.patch.is_binary` function. 

56 

57The binary detection heuristic implementation is close to the one in JGit: 

58https://github.com/eclipse/jgit/blob/f6873ffe522bbc3536969a3a3546bf9a819b92bf/org.eclipse.jgit/src/org/eclipse/jgit/diff/RawText.java#L300 

59 

60There is multiple variables that impact the normalization. 

61 

62First, a repository can contains a ``.gitattributes`` file (or more than one...) 

63that can further customize the operation on some file patterns, for example: 

64 

65 \*.txt text 

66 

67Force all ``.txt`` files to be treated as text files and to have their lines 

68endings normalized. 

69 

70 \*.jpg -text 

71 

72Force all ``.jpg`` files to be treated as binary files and to not have their 

73lines endings converted. 

74 

75 \*.vcproj text eol=crlf 

76 

77Force all ``.vcproj`` files to be treated as text files and to have their lines 

78endings converted into ``CRLF`` in working directory no matter the native EOL of 

79the platform. 

80 

81 \*.sh text eol=lf 

82 

83Force all ``.sh`` files to be treated as text files and to have their lines 

84endings converted into ``LF`` in working directory no matter the native EOL of 

85the platform. 

86 

87If the ``eol`` attribute is not defined, Git uses the ``core.eol`` configuration 

88value described later. 

89 

90 \* text=auto 

91 

92Force all files to be scanned by the text file heuristic detection and to have 

93their line endings normalized in case they are detected as text files. 

94 

95Git also have a obsolete attribute named ``crlf`` that can be translated to the 

96corresponding text attribute value. 

97 

98Then there are some configuration option (that can be defined at the 

99repository or user level): 

100 

101- core.autocrlf 

102- core.eol 

103 

104``core.autocrlf`` is taken into account for all files that doesn't have a ``text`` 

105attribute defined in ``.gitattributes``; it takes three possible values: 

106 

107 - ``true``: This forces all files on the working directory to have CRLF 

108 line-endings in the working directory and convert line-endings to LF 

109 when writing to the index. An explicit ``eol`` attribute still takes 

110 precedence over it. 

111 - ``input``: Quite similar to the ``true`` value but only applies the clean 

112 filter, ie line-ending of new files added to the index will get their 

113 line-endings converted to LF. 

114 - ``false`` (default): No normalization is done. 

115 

116``core.eol`` is the top-level configuration to define the line-ending to use 

117when applying the smudge filter. It takes three possible values: 

118 

119 - ``lf``: When normalization is done, force line-endings to be ``LF`` in the 

120 working directory. 

121 - ``crlf``: When normalization is done, force line-endings to be ``CRLF`` in 

122 the working directory. 

123 - ``native`` (default): When normalization is done, force line-endings to be 

124 the platform's native line ending. 

125 

126One thing to remember is when line-ending normalization is done on a file, Git 

127always normalize line-ending to ``LF`` when writing to the index. 

128 

129There are sources that seems to indicate that Git won't do line-ending 

130normalization when a file contains mixed line-endings. I think this logic 

131might be in text / binary detection heuristic but couldn't find it yet. 

132 

133Sources: 

134- https://git-scm.com/docs/git-config#git-config-coreeol 

135- https://git-scm.com/docs/git-config#git-config-coreautocrlf 

136- https://git-scm.com/docs/gitattributes#_checking_out_and_checking_in 

137- https://adaptivepatchwork.com/2012/03/01/mind-the-end-of-your-line/ 

138""" 

139 

140__all__ = [ 

141 "CRLF", 

142 "LF", 

143 "BlobNormalizer", 

144 "CRLFAction", 

145 "LineEndingFilter", 

146 "TreeBlobNormalizer", 

147 "check_safecrlf", 

148 "convert_crlf_to_lf", 

149 "convert_lf_to_crlf", 

150 "get_clean_filter", 

151 "get_clean_filter_autocrlf", 

152 "get_smudge_filter", 

153 "get_smudge_filter_autocrlf", 

154 "line_ending_filter_for_action", 

155 "normalize_blob", 

156 "read_eol_config", 

157 "resolve_crlf_action", 

158] 

159 

160import logging 

161import os 

162from collections.abc import Callable, Mapping 

163from enum import Enum 

164from typing import TYPE_CHECKING, Any 

165 

166if TYPE_CHECKING: 

167 from .config import Config 

168 from .object_store import BaseObjectStore 

169 

170from .attrs import AttributeValue, GitAttributes, Pattern 

171from .filters import FilterBlobNormalizer, FilterContext, FilterRegistry 

172from .object_store import iter_tree_contents 

173from .objects import Blob, ObjectID 

174from .patch import is_binary 

175from .wildmatch import MalformedPattern 

176 

177CRLF = b"\r\n" 

178LF = b"\n" 

179 

180logger = logging.getLogger(__name__) 

181 

182 

183def read_eol_config(config: "Config | None") -> tuple[str, bytes, bytes]: 

184 """Read the line ending related settings from a configuration. 

185 

186 Returns: Tuple with the values of ``core.eol``, ``core.autocrlf`` and 

187 ``core.safecrlf`` 

188 """ 

189 if config is None: 

190 return "native", b"false", b"false" 

191 

192 try: 

193 core_eol_raw = config.get("core", "eol") 

194 except KeyError: 

195 core_eol = "native" 

196 else: 

197 core_eol = ( 

198 core_eol_raw.decode("ascii") 

199 if isinstance(core_eol_raw, bytes) 

200 else str(core_eol_raw) 

201 ).lower() 

202 

203 try: 

204 autocrlf_raw = config.get("core", "autocrlf") 

205 except KeyError: 

206 autocrlf = b"false" 

207 else: 

208 autocrlf = ( 

209 autocrlf_raw.lower() 

210 if isinstance(autocrlf_raw, bytes) 

211 else str(autocrlf_raw).lower().encode("ascii") 

212 ) 

213 

214 try: 

215 safecrlf_raw = config.get("core", "safecrlf") 

216 except KeyError: 

217 safecrlf = b"false" 

218 else: 

219 safecrlf = ( 

220 safecrlf_raw 

221 if isinstance(safecrlf_raw, bytes) 

222 else safecrlf_raw.encode("utf-8") 

223 ) 

224 

225 return core_eol, autocrlf, safecrlf 

226 

227 

228class CRLFAction(Enum): 

229 """The line ending conversion to apply to a path. 

230 

231 This mirrors git's ``enum convert_crlf_action`` in convert.c. 

232 ``UNDEFINED``, ``TEXT`` and ``AUTO`` are intermediate states only; 

233 :func:`resolve_crlf_action` always resolves them into one of the 

234 others. 

235 """ 

236 

237 UNDEFINED = "undefined" 

238 BINARY = "binary" 

239 TEXT = "text" 

240 TEXT_INPUT = "text-input" 

241 TEXT_CRLF = "text-crlf" 

242 AUTO = "auto" 

243 AUTO_INPUT = "auto-input" 

244 AUTO_CRLF = "auto-crlf" 

245 

246 

247# Actions where git applies its text/binary heuristic rather than trusting the 

248# attributes, and so refuses to touch files that already contain CR. 

249_AUTO_ACTIONS = (CRLFAction.AUTO, CRLFAction.AUTO_INPUT, CRLFAction.AUTO_CRLF) 

250 

251 

252def _crlf_action_from_attr(value: AttributeValue) -> CRLFAction: 

253 """Translate a ``text`` (or obsolete ``crlf``) attribute into an action.""" 

254 if value is True: 

255 return CRLFAction.TEXT 

256 elif value is False: 

257 return CRLFAction.BINARY 

258 elif value == b"input": 

259 return CRLFAction.TEXT_INPUT 

260 elif value == b"auto": 

261 return CRLFAction.AUTO 

262 return CRLFAction.UNDEFINED 

263 

264 

265def _text_eol_is_crlf(core_eol: str, autocrlf: bytes) -> bool: 

266 """Decide the working tree line ending when only the config says so.""" 

267 if autocrlf == b"true": 

268 return True 

269 elif autocrlf == b"input": 

270 return False 

271 if core_eol == "crlf": 

272 return True 

273 if core_eol == "native": 

274 return os.linesep == "\r\n" 

275 return False 

276 

277 

278def resolve_crlf_action( 

279 attributes: Mapping[bytes, AttributeValue], 

280 core_eol: str = "native", 

281 autocrlf: bytes = b"false", 

282) -> CRLFAction: 

283 """Determine the line ending conversion for a path. 

284 

285 This follows ``convert_attrs`` in git's convert.c: the ``text`` attribute 

286 (or the obsolete ``crlf`` one) selects whether conversion happens at all, 

287 an explicit ``eol`` attribute then pins the working tree line ending, and 

288 ``core.autocrlf``/``core.eol`` only apply where the attributes are silent. 

289 

290 Args: 

291 attributes: Attributes matched for the path 

292 core_eol: Value of ``core.eol`` 

293 autocrlf: Value of ``core.autocrlf`` 

294 

295 Returns: The action to apply 

296 """ 

297 action = _crlf_action_from_attr(attributes.get(b"text")) 

298 if action == CRLFAction.UNDEFINED: 

299 action = _crlf_action_from_attr(attributes.get(b"crlf")) 

300 

301 if action != CRLFAction.BINARY: 

302 eol_attr = attributes.get(b"eol") 

303 if action == CRLFAction.AUTO and eol_attr == b"lf": 

304 action = CRLFAction.AUTO_INPUT 

305 elif action == CRLFAction.AUTO and eol_attr == b"crlf": 

306 action = CRLFAction.AUTO_CRLF 

307 elif eol_attr == b"lf": 

308 action = CRLFAction.TEXT_INPUT 

309 elif eol_attr == b"crlf": 

310 action = CRLFAction.TEXT_CRLF 

311 

312 if action in (CRLFAction.TEXT, CRLFAction.AUTO): 

313 # Neither the attributes nor an "eol" attribute pinned the working 

314 # tree line ending, so fall back to the configuration. 

315 crlf = _text_eol_is_crlf(core_eol, autocrlf) 

316 if action == CRLFAction.TEXT: 

317 action = CRLFAction.TEXT_CRLF if crlf else CRLFAction.TEXT_INPUT 

318 else: 

319 action = CRLFAction.AUTO_CRLF if crlf else CRLFAction.AUTO_INPUT 

320 if action == CRLFAction.UNDEFINED: 

321 if autocrlf == b"true": 

322 action = CRLFAction.AUTO_CRLF 

323 elif autocrlf == b"input": 

324 action = CRLFAction.AUTO_INPUT 

325 else: 

326 action = CRLFAction.BINARY 

327 

328 return action 

329 

330 

331def line_ending_filter_for_action( 

332 action: CRLFAction, safecrlf: bytes = b"false" 

333) -> "LineEndingFilter | None": 

334 """Build the filter implementing a resolved :class:`CRLFAction`. 

335 

336 Args: 

337 action: Action as returned by :func:`resolve_crlf_action` 

338 safecrlf: Value of ``core.safecrlf`` 

339 

340 Returns: A filter, or None if the action converts nothing 

341 """ 

342 if action == CRLFAction.BINARY: 

343 return None 

344 

345 smudge: Callable[[bytes], bytes] | None = ( 

346 convert_lf_to_crlf 

347 if action in (CRLFAction.TEXT_CRLF, CRLFAction.AUTO_CRLF) 

348 else None 

349 ) 

350 return LineEndingFilter( 

351 clean_conversion=convert_crlf_to_lf, 

352 smudge_conversion=smudge, 

353 binary_detection=True, 

354 safecrlf=safecrlf, 

355 # Only the "auto" family leaves files with existing CRs alone; an 

356 # explicit "text" attribute means the user has already decided. 

357 require_lf_only=action in _AUTO_ACTIONS, 

358 ) 

359 

360 

361class LineEndingFilter: 

362 """Filter driver for line ending conversion. 

363 

364 Satisfies the :class:`dulwich.filters.FilterDriver` protocol structurally, 

365 without inheriting from it, so that the line ending logic this module 

366 exports can be imported from ``filters`` without a circular import. 

367 """ 

368 

369 def __init__( 

370 self, 

371 clean_conversion: Callable[[bytes], bytes] | None = None, 

372 smudge_conversion: Callable[[bytes], bytes] | None = None, 

373 binary_detection: bool = True, 

374 safecrlf: bytes = b"false", 

375 require_lf_only: bool = False, 

376 ): 

377 """Initialize LineEndingFilter. 

378 

379 Args: 

380 clean_conversion: Conversion to apply on checkin 

381 smudge_conversion: Conversion to apply on checkout 

382 binary_detection: Skip files detected as binary 

383 safecrlf: Value of ``core.safecrlf`` 

384 require_lf_only: Only smudge content whose line endings are all LF, 

385 as git does for ``text=auto`` and ``core.autocrlf`` 

386 """ 

387 self.clean_conversion = clean_conversion 

388 self.smudge_conversion = smudge_conversion 

389 self.binary_detection = binary_detection 

390 self.safecrlf = safecrlf 

391 self.require_lf_only = require_lf_only 

392 

393 @classmethod 

394 def from_config( 

395 cls, config: "Config | None", for_text_attr: bool = False 

396 ) -> "LineEndingFilter": 

397 """Create a LineEndingFilter from git configuration. 

398 

399 Args: 

400 config: Git configuration 

401 for_text_attr: If True, always normalize on checkin (for text attribute) 

402 

403 Returns: 

404 Configured LineEndingFilter instance 

405 """ 

406 if config is None: 

407 # Default filter 

408 if for_text_attr: 

409 # For text attribute: always normalize on checkin 

410 return cls( 

411 clean_conversion=convert_crlf_to_lf, 

412 smudge_conversion=None, 

413 binary_detection=True, 

414 ) 

415 else: 

416 # No config: no conversion 

417 return cls() 

418 

419 core_eol, autocrlf, safecrlf = read_eol_config(config) 

420 

421 if for_text_attr: 

422 # For text attribute: always normalize to LF on checkin 

423 # Smudge behavior depends on core.eol and core.autocrlf 

424 smudge_filter = get_smudge_filter(core_eol, autocrlf) 

425 clean_filter: Callable[[bytes], bytes] | None = convert_crlf_to_lf 

426 else: 

427 # Normal autocrlf behavior 

428 smudge_filter = get_smudge_filter(core_eol, autocrlf) 

429 clean_filter = get_clean_filter(core_eol, autocrlf) 

430 

431 return cls( 

432 clean_conversion=clean_filter, 

433 smudge_conversion=smudge_filter, 

434 binary_detection=True, 

435 safecrlf=safecrlf, 

436 ) 

437 

438 def clean(self, data: bytes, path: bytes = b"") -> bytes: 

439 """Apply line ending conversion for checkin (working tree -> repository).""" 

440 if self.clean_conversion is None: 

441 return data 

442 

443 # Skip binary files if detection is enabled 

444 if self.binary_detection and is_binary(data): 

445 return data 

446 

447 converted = self.clean_conversion(data) 

448 

449 # Check if conversion is safe 

450 if self.safecrlf != b"false": 

451 check_safecrlf(data, converted, self.safecrlf, path) 

452 

453 return converted 

454 

455 def smudge(self, data: bytes, path: bytes = b"") -> bytes: 

456 """Apply line ending conversion for checkout (repository -> working tree).""" 

457 if self.smudge_conversion is None: 

458 return data 

459 

460 # Skip binary files if detection is enabled 

461 if self.binary_detection and is_binary(data): 

462 return data 

463 

464 # Git leaves content that already contains CR alone when the decision 

465 # came from a heuristic rather than an explicit attribute. 

466 if self.require_lf_only and b"\r" in data: 

467 return data 

468 

469 converted = self.smudge_conversion(data) 

470 

471 # Check if conversion is safe 

472 if self.safecrlf != b"false": 

473 check_safecrlf(data, converted, self.safecrlf, path) 

474 

475 return converted 

476 

477 def cleanup(self) -> None: 

478 """Clean up any resources held by this filter driver.""" 

479 # LineEndingFilter doesn't hold any resources that need cleanup 

480 

481 def reuse(self, config: "Config", filter_name: str) -> bool: 

482 """Check if this filter driver should be reused with the given configuration.""" 

483 # LineEndingFilter is lightweight and should always be recreated 

484 # to ensure it uses the latest configuration 

485 return False 

486 

487 

488def convert_crlf_to_lf(text_hunk: bytes) -> bytes: 

489 """Convert CRLF in text hunk into LF. 

490 

491 Args: 

492 text_hunk: A bytes string representing a text hunk 

493 Returns: The text hunk with the same type, with CRLF replaced into LF 

494 """ 

495 return text_hunk.replace(CRLF, LF) 

496 

497 

498def convert_lf_to_crlf(text_hunk: bytes) -> bytes: 

499 """Convert LF in text hunk into CRLF. 

500 

501 Args: 

502 text_hunk: A bytes string representing a text hunk 

503 Returns: The text hunk with the same type, with LF replaced into CRLF 

504 """ 

505 # Single-pass conversion: split on LF and join with CRLF 

506 # This avoids the double replacement issue 

507 parts = text_hunk.split(LF) 

508 # Remove any trailing CR to avoid CRCRLF 

509 cleaned_parts = [] 

510 for i, part in enumerate(parts): 

511 if i < len(parts) - 1 and part.endswith(b"\r"): 

512 cleaned_parts.append(part[:-1]) 

513 else: 

514 cleaned_parts.append(part) 

515 return CRLF.join(cleaned_parts) 

516 

517 

518def check_safecrlf( 

519 original: bytes, converted: bytes, safecrlf: bytes, path: bytes = b"" 

520) -> None: 

521 """Check if CRLF conversion is safe according to core.safecrlf setting. 

522 

523 Args: 

524 original: Original content before conversion 

525 converted: Content after conversion 

526 safecrlf: Value of core.safecrlf config (b"true", b"warn", or b"false") 

527 path: Path to the file being checked (for error messages) 

528 

529 Raises: 

530 ValueError: If safecrlf is "true" and conversion would lose data 

531 """ 

532 if safecrlf == b"false": 

533 return 

534 

535 # Check if conversion is reversible 

536 if safecrlf in (b"true", b"warn"): 

537 # For CRLF->LF conversion, check if converting back would recover original 

538 if CRLF in original and CRLF not in converted: 

539 # This was a CRLF->LF conversion 

540 recovered = convert_lf_to_crlf(converted) 

541 if recovered != original: 

542 msg = ( 

543 f"CRLF would be replaced by LF in {path.decode('utf-8', 'replace')}" 

544 ) 

545 if safecrlf == b"true": 

546 raise ValueError(msg) 

547 else: # warn 

548 logger.warning(msg) 

549 

550 # For LF->CRLF conversion, check if converting back would recover original 

551 elif LF in original and CRLF in converted and CRLF not in original: 

552 # This was a LF->CRLF conversion 

553 recovered = convert_crlf_to_lf(converted) 

554 if recovered != original: 

555 msg = ( 

556 f"LF would be replaced by CRLF in {path.decode('utf-8', 'replace')}" 

557 ) 

558 if safecrlf == b"true": 

559 raise ValueError(msg) 

560 else: # warn 

561 logger.warning(msg) 

562 

563 

564def get_smudge_filter( 

565 core_eol: str, core_autocrlf: bytes 

566) -> Callable[[bytes], bytes] | None: 

567 """Returns the correct smudge filter based on the passed arguments.""" 

568 # Git attributes handling is done by the filter infrastructure 

569 return get_smudge_filter_autocrlf(core_autocrlf) 

570 

571 

572def get_clean_filter( 

573 core_eol: str, core_autocrlf: bytes 

574) -> Callable[[bytes], bytes] | None: 

575 """Returns the correct clean filter based on the passed arguments.""" 

576 # Git attributes handling is done by the filter infrastructure 

577 return get_clean_filter_autocrlf(core_autocrlf) 

578 

579 

580def get_smudge_filter_autocrlf( 

581 core_autocrlf: bytes, 

582) -> Callable[[bytes], bytes] | None: 

583 """Returns the correct smudge filter base on autocrlf value. 

584 

585 Args: 

586 core_autocrlf: The bytes configuration value of core.autocrlf. 

587 Valid values are: b'true', b'false' or b'input'. 

588 Returns: Either None if no filter has to be applied or a function 

589 accepting a single argument, a binary text hunk 

590 """ 

591 if core_autocrlf == b"true": 

592 return convert_lf_to_crlf 

593 

594 return None 

595 

596 

597def get_clean_filter_autocrlf( 

598 core_autocrlf: bytes, 

599) -> Callable[[bytes], bytes] | None: 

600 """Returns the correct clean filter base on autocrlf value. 

601 

602 Args: 

603 core_autocrlf: The bytes configuration value of core.autocrlf. 

604 Valid values are: b'true', b'false' or b'input'. 

605 Returns: Either None if no filter has to be applied or a function 

606 accepting a single argument, a binary text hunk 

607 """ 

608 if core_autocrlf == b"true" or core_autocrlf == b"input": 

609 return convert_crlf_to_lf 

610 

611 # Checking filter should never be `convert_lf_to_crlf` 

612 return None 

613 

614 

615class BlobNormalizer(FilterBlobNormalizer): 

616 """An object to store computation result of which filter to apply based on configuration, gitattributes, path and operation (checkin or checkout). 

617 

618 This class maintains backward compatibility while using the filter infrastructure. 

619 """ 

620 

621 def __init__( 

622 self, 

623 config_stack: "Config", 

624 gitattributes: Mapping[str, Any], 

625 core_eol: str = "native", 

626 autocrlf: bytes = b"false", 

627 safecrlf: bytes = b"false", 

628 ) -> None: 

629 """Initialize FilteringBlobNormalizer.""" 

630 # Set up a filter registry with line ending filters 

631 filter_registry = FilterRegistry(config_stack) 

632 

633 # Create line ending filter if needed 

634 smudge_filter = get_smudge_filter(core_eol, autocrlf) 

635 clean_filter = get_clean_filter(core_eol, autocrlf) 

636 

637 # Always register a text filter that can be used by gitattributes 

638 # Even if autocrlf is false, gitattributes text=true should work 

639 line_ending_filter = LineEndingFilter( 

640 clean_conversion=clean_filter or convert_crlf_to_lf, 

641 smudge_conversion=smudge_filter or convert_lf_to_crlf, 

642 binary_detection=True, 

643 safecrlf=safecrlf, 

644 ) 

645 filter_registry.register_driver("text", line_ending_filter) 

646 

647 # Convert dict gitattributes to GitAttributes object for parent class 

648 git_attrs_patterns = [] 

649 for pattern_str, attrs in gitattributes.items(): 

650 if isinstance(pattern_str, str): 

651 pattern_bytes = pattern_str.encode("utf-8") 

652 else: 

653 pattern_bytes = pattern_str 

654 try: 

655 pattern = Pattern(pattern_bytes) 

656 except MalformedPattern: 

657 logger.warning("Ignoring malformed pattern %r", pattern_bytes) 

658 continue 

659 git_attrs_patterns.append((pattern, attrs)) 

660 

661 git_attributes = GitAttributes(git_attrs_patterns) 

662 

663 # Create FilterContext for parent class 

664 filter_context = FilterContext(filter_registry) 

665 

666 # Initialize parent class with gitattributes 

667 # The filter infrastructure will handle gitattributes processing 

668 super().__init__(config_stack, git_attributes, filter_context=filter_context) 

669 

670 # Store original filters for backward compatibility 

671 self.fallback_read_filter = smudge_filter 

672 self.fallback_write_filter = clean_filter 

673 

674 def checkin_normalize(self, blob: Blob, path: bytes) -> Blob: 

675 """Normalize a blob during a checkin operation.""" 

676 # First try to get filter from gitattributes (handled by parent) 

677 result = super().checkin_normalize(blob, path) 

678 

679 # Check if gitattributes explicitly disabled text conversion 

680 attrs = self.gitattributes.match_path(path) 

681 if b"text" in attrs and attrs[b"text"] is False: 

682 # Explicitly marked as binary, no conversion 

683 return blob 

684 

685 # If no filter was applied via gitattributes and we have a fallback filter 

686 # (autocrlf is enabled), apply it to all files 

687 if result is blob and self.fallback_write_filter is not None: 

688 # Apply the clean filter with binary detection 

689 # Get safecrlf from config 

690 safecrlf = b"false" 

691 config_stack = getattr(self.filter_registry, "config_stack", None) 

692 if config_stack is not None: 

693 safecrlf = config_stack.get(b"core", b"safecrlf", b"false") 

694 if hasattr(safecrlf, "encode"): 

695 safecrlf = safecrlf.encode("utf-8") 

696 

697 line_ending_filter = LineEndingFilter( 

698 clean_conversion=self.fallback_write_filter, 

699 smudge_conversion=None, 

700 binary_detection=True, 

701 safecrlf=safecrlf, 

702 ) 

703 filtered_data = line_ending_filter.clean(blob.data, path) 

704 if filtered_data != blob.data: 

705 new_blob = Blob() 

706 new_blob.data = filtered_data 

707 return new_blob 

708 

709 return result 

710 

711 def checkout_normalize(self, blob: Blob, path: bytes) -> Blob: 

712 """Normalize a blob during a checkout operation.""" 

713 # First try to get filter from gitattributes (handled by parent) 

714 result = super().checkout_normalize(blob, path) 

715 

716 # Check if gitattributes explicitly disabled text conversion 

717 attrs = self.gitattributes.match_path(path) 

718 if b"text" in attrs and attrs[b"text"] is False: 

719 # Explicitly marked as binary, no conversion 

720 return blob 

721 

722 # If no filter was applied via gitattributes and we have a fallback filter 

723 # (autocrlf is enabled), apply it to all files 

724 if result is blob and self.fallback_read_filter is not None: 

725 # Apply the smudge filter with binary detection 

726 # Get safecrlf from config 

727 safecrlf = b"false" 

728 config_stack = getattr(self.filter_registry, "config_stack", None) 

729 if config_stack is not None: 

730 safecrlf = config_stack.get(b"core", b"safecrlf", b"false") 

731 if hasattr(safecrlf, "encode"): 

732 safecrlf = safecrlf.encode("utf-8") 

733 

734 line_ending_filter = LineEndingFilter( 

735 clean_conversion=None, 

736 smudge_conversion=self.fallback_read_filter, 

737 binary_detection=True, 

738 safecrlf=safecrlf, 

739 ) 

740 filtered_data = line_ending_filter.smudge(blob.data, path) 

741 if filtered_data != blob.data: 

742 new_blob = Blob() 

743 new_blob.data = filtered_data 

744 return new_blob 

745 

746 return result 

747 

748 

749def normalize_blob( 

750 blob: Blob, conversion: Callable[[bytes], bytes], binary_detection: bool 

751) -> Blob: 

752 """Normalize blob by applying line ending conversion.""" 

753 # Read the original blob 

754 data = blob.data 

755 

756 # If we need to detect if a file is binary and the file is detected as 

757 # binary, do not apply the conversion function and return the original 

758 # chunked text 

759 if binary_detection is True: 

760 if is_binary(data): 

761 return blob 

762 

763 # Now apply the conversion 

764 converted_data = conversion(data) 

765 

766 new_blob = Blob() 

767 new_blob.data = converted_data 

768 

769 return new_blob 

770 

771 

772class TreeBlobNormalizer(BlobNormalizer): 

773 """Blob normalizer that tracks existing files in a tree.""" 

774 

775 def __init__( 

776 self, 

777 config_stack: "Config", 

778 git_attributes: Mapping[str, Any], 

779 object_store: "BaseObjectStore", 

780 tree: ObjectID | None = None, 

781 core_eol: str = "native", 

782 autocrlf: bytes = b"false", 

783 safecrlf: bytes = b"false", 

784 ) -> None: 

785 """Initialize TreeBlobNormalizer.""" 

786 super().__init__(config_stack, git_attributes, core_eol, autocrlf, safecrlf) 

787 if tree: 

788 self.existing_paths = { 

789 name for name, _, _ in iter_tree_contents(object_store, tree) 

790 } 

791 else: 

792 self.existing_paths = set() 

793 

794 def checkin_normalize(self, blob: Blob, path: bytes) -> Blob: 

795 """Normalize blob for checkin, considering existing tree state.""" 

796 # Existing files should only be normalized on checkin if: 

797 # 1. They were previously normalized on checkout (autocrlf=true), OR 

798 # 2. We have a write filter (autocrlf=true or autocrlf=input), OR 

799 # 3. They are new files 

800 if ( 

801 self.fallback_read_filter is not None 

802 or self.fallback_write_filter is not None 

803 or path not in self.existing_paths 

804 ): 

805 return super().checkin_normalize(blob, path) 

806 return blob