Coverage for /pythoncovmergedfiles/medio/medio/usr/local/lib/python3.8/site-packages/ftfy/bad_codecs/__init__.py: 95%

1r""" 

2The `ftfy.bad_codecs` module gives Python the ability to decode some common, 

3flawed encodings. 

4 

5Python does not want you to be sloppy with your text. Its encoders and decoders 

6("codecs") follow the relevant standards whenever possible, which means that 

7when you get text that *doesn't* follow those standards, you'll probably fail 

8to decode it. Or you might succeed at decoding it for implementation-specific 

9reasons, which is perhaps worse. 

10 

11There are some encodings out there that Python wishes didn't exist, which are 

12widely used outside of Python: 

13 

14- "utf-8-variants", a family of not-quite-UTF-8 encodings, including the 

15 ever-popular CESU-8 and "Java modified UTF-8". 

16- "Sloppy" versions of character map encodings, where bytes that don't map to 

17 anything will instead map to the Unicode character with the same number. 

18 

19Simply importing this module, or in fact any part of the `ftfy` package, will 

20make these new "bad codecs" available to Python through the standard Codecs 

21API. You never have to actually call any functions inside `ftfy.bad_codecs`. 

22 

23However, if you want to call something because your code checker insists on it, 

24you can call ``ftfy.bad_codecs.ok()``. 

25 

26A quick example of decoding text that's encoded in CESU-8: 

27 

28 >>> import ftfy.bad_codecs 

29 >>> print(b'\xed\xa0\xbd\xed\xb8\x8d'.decode('utf-8-variants')) 

30 😍 

31""" 

32from encodings import normalize_encoding 

33import codecs 

34from typing import Dict 

35 

36_CACHE: Dict[str, codecs.CodecInfo] = {} 

37 

38# Define some aliases for 'utf-8-variants'. All hyphens get turned into 

39# underscores, because of `normalize_encoding`. 

40UTF8_VAR_NAMES = ( 

41 "utf_8_variants", 

42 "utf8_variants", 

43 "utf_8_variant", 

44 "utf8_variant", 

45 "utf_8_var", 

46 "utf8_var", 

47 "cesu_8", 

48 "cesu8", 

49 "java_utf_8", 

50 "java_utf8", 

51) 

52 

53 

54def search_function(encoding): 

55 """ 

56 Register our "bad codecs" with Python's codecs API. This involves adding 

57 a search function that takes in an encoding name, and returns a codec 

58 for that encoding if it knows one, or None if it doesn't. 

59 

60 The encodings this will match are: 

61 

62 - Encodings of the form 'sloppy-windows-NNNN' or 'sloppy-iso-8859-N', 

63 where the non-sloppy version is an encoding that leaves some bytes 

64 unmapped to characters. 

65 - The 'utf-8-variants' encoding, which has the several aliases seen 

66 above. 

67 """ 

68 if encoding in _CACHE: 

69 return _CACHE[encoding] 

70 

71 norm_encoding = normalize_encoding(encoding) 

72 codec = None 

73 if norm_encoding in UTF8_VAR_NAMES: 

74 from ftfy.bad_codecs.utf8_variants import CODEC_INFO 

75 

76 codec = CODEC_INFO 

77 elif norm_encoding.startswith("sloppy_"): 

78 from ftfy.bad_codecs.sloppy import CODECS 

79 

80 codec = CODECS.get(norm_encoding) 

81 

82 if codec is not None: 

83 _CACHE[encoding] = codec 

84 

85 return codec 

86 

87 

88def ok(): 

89 """ 

90 A feel-good function that gives you something to call after importing 

91 this package. 

92 

93 Why is this here? Pyflakes. Pyflakes gets upset when you import a module 

94 and appear not to use it. It doesn't know that you're using it when 

95 you use the ``unicode.encode`` and ``bytes.decode`` methods with certain 

96 encodings. 

97 """ 

98 

99 

100codecs.register(search_function)