Coverage for /pythoncovmergedfiles/medio/medio/usr/local/lib/python3.8/site-packages/keras/src/layers/preprocessing/integer_lookup.py: 31%

36 statements  

« prev     ^ index     » next       coverage.py v7.4.0, created at 2024-01-03 07:57 +0000

1# Copyright 2020 The TensorFlow Authors. All Rights Reserved. 

2# 

3# Licensed under the Apache License, Version 2.0 (the "License"); 

4# you may not use this file except in compliance with the License. 

5# You may obtain a copy of the License at 

6# 

7# http://www.apache.org/licenses/LICENSE-2.0 

8# 

9# Unless required by applicable law or agreed to in writing, software 

10# distributed under the License is distributed on an "AS IS" BASIS, 

11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 

12# See the License for the specific language governing permissions and 

13# limitations under the License. 

14# ============================================================================== 

15"""Keras string lookup preprocessing layer.""" 

16 

17 

18import numpy as np 

19import tensorflow.compat.v2 as tf 

20 

21from keras.src.engine import base_preprocessing_layer 

22from keras.src.layers.preprocessing import index_lookup 

23 

24# isort: off 

25from tensorflow.python.platform import tf_logging as logging 

26from tensorflow.python.util.tf_export import keras_export 

27 

28 

29@keras_export( 

30 "keras.layers.IntegerLookup", 

31 "keras.layers.experimental.preprocessing.IntegerLookup", 

32 v1=[], 

33) 

34class IntegerLookup(index_lookup.IndexLookup): 

35 """A preprocessing layer which maps integer features to contiguous ranges. 

36 

37 This layer maps a set of arbitrary integer input tokens into indexed integer 

38 output via a table-based vocabulary lookup. The layer's output indices will 

39 be contiguously arranged up to the maximum vocab size, even if the input 

40 tokens are non-continguous or unbounded. The layer supports multiple options 

41 for encoding the output via `output_mode`, and has optional support for 

42 out-of-vocabulary (OOV) tokens and masking. 

43 

44 The vocabulary for the layer must be either supplied on construction or 

45 learned via `adapt()`. During `adapt()`, the layer will analyze a data set, 

46 determine the frequency of individual integer tokens, and create a 

47 vocabulary from them. If the vocabulary is capped in size, the most frequent 

48 tokens will be used to create the vocabulary and all others will be treated 

49 as OOV. 

50 

51 There are two possible output modes for the layer. When `output_mode` is 

52 `"int"`, input integers are converted to their index in the vocabulary (an 

53 integer). When `output_mode` is `"multi_hot"`, `"count"`, or `"tf_idf"`, 

54 input integers are encoded into an array where each dimension corresponds to 

55 an element in the vocabulary. 

56 

57 The vocabulary can optionally contain a mask token as well as an OOV token 

58 (which can optionally occupy multiple indices in the vocabulary, as set 

59 by `num_oov_indices`). 

60 The position of these tokens in the vocabulary is fixed. When `output_mode` 

61 is `"int"`, the vocabulary will begin with the mask token at index 0, 

62 followed by OOV indices, followed by the rest of the vocabulary. When 

63 `output_mode` is `"multi_hot"`, `"count"`, or `"tf_idf"` the vocabulary will 

64 begin with OOV indices and instances of the mask token will be dropped. 

65 

66 For an overview and full list of preprocessing layers, see the preprocessing 

67 [guide](https://www.tensorflow.org/guide/keras/preprocessing_layers). 

68 

69 Args: 

70 max_tokens: Maximum size of the vocabulary for this layer. This should 

71 only be specified when adapting the vocabulary or when setting 

72 `pad_to_max_tokens=True`. If None, there is no cap on the size of the 

73 vocabulary. Note that this size includes the OOV and mask tokens. 

74 Defaults to `None`. 

75 num_oov_indices: The number of out-of-vocabulary tokens to use. If this 

76 value is more than 1, OOV inputs are modulated to determine their OOV 

77 value. If this value is 0, OOV inputs will cause an error when calling 

78 the layer. Defaults to `1`. 

79 mask_token: An integer token that represents masked inputs. When 

80 `output_mode` is `"int"`, the token is included in vocabulary and mapped 

81 to index 0. In other output modes, the token will not appear in the 

82 vocabulary and instances of the mask token in the input will be dropped. 

83 If set to None, no mask term will be added. Defaults to `None`. 

84 oov_token: Only used when `invert` is True. The token to return for OOV 

85 indices. Defaults to `-1`. 

86 vocabulary: Optional. Either an array of integers or a string path to a 

87 text file. If passing an array, can pass a tuple, list, 1D numpy array, 

88 or 1D tensor containing the integer vocbulary terms. If passing a file 

89 path, the file should contain one line per term in the vocabulary. If 

90 this argument is set, there is no need to `adapt()` the layer. 

91 vocabulary_dtype: The dtype of the vocabulary terms, for example 

92 `"int64"` or `"int32"`. Defaults to `"int64"`. 

93 idf_weights: Only valid when `output_mode` is `"tf_idf"`. A tuple, list, 

94 1D numpy array, or 1D tensor or the same length as the vocabulary, 

95 containing the floating point inverse document frequency weights, which 

96 will be multiplied by per sample term counts for the final `tf_idf` 

97 weight. If the `vocabulary` argument is set, and `output_mode` is 

98 `"tf_idf"`, this argument must be supplied. 

99 invert: Only valid when `output_mode` is `"int"`. If True, this layer will 

100 map indices to vocabulary items instead of mapping vocabulary items to 

101 indices. Defaults to `False`. 

102 output_mode: Specification for the output of the layer. Values can be 

103 `"int"`, `"one_hot"`, `"multi_hot"`, `"count"`, or `"tf_idf"` 

104 configuring the layer as follows: 

105 - `"int"`: Return the vocabulary indices of the input tokens. 

106 - `"one_hot"`: Encodes each individual element in the input into an 

107 array the same size as the vocabulary, containing a 1 at the element 

108 index. If the last dimension is size 1, will encode on that 

109 dimension. If the last dimension is not size 1, will append a new 

110 dimension for the encoded output. 

111 - `"multi_hot"`: Encodes each sample in the input into a single array 

112 the same size as the vocabulary, containing a 1 for each vocabulary 

113 term present in the sample. Treats the last dimension as the sample 

114 dimension, if input shape is (..., sample_length), output shape will 

115 be (..., num_tokens). 

116 - `"count"`: As `"multi_hot"`, but the int array contains a count of 

117 the number of times the token at that index appeared in the sample. 

118 - `"tf_idf"`: As `"multi_hot"`, but the TF-IDF algorithm is applied to 

119 find the value in each token slot. 

120 For `"int"` output, any shape of input and output is supported. For all 

121 other output modes, currently only output up to rank 2 is supported. 

122 Defaults to `"int"`. 

123 pad_to_max_tokens: Only applicable when `output_mode` is `"multi_hot"`, 

124 `"count"`, or `"tf_idf"`. If True, the output will have its feature axis 

125 padded to `max_tokens` even if the number of unique tokens in the 

126 vocabulary is less than max_tokens, resulting in a tensor of shape 

127 [batch_size, max_tokens] regardless of vocabulary size. Defaults to 

128 False. 

129 sparse: Boolean. Only applicable when `output_mode` is `"multi_hot"`, 

130 `"count"`, or `"tf_idf"`. If True, returns a `SparseTensor` instead of a 

131 dense `Tensor`. Defaults to `False`. 

132 

133 Examples: 

134 

135 **Creating a lookup layer with a known vocabulary** 

136 

137 This example creates a lookup layer with a pre-existing vocabulary. 

138 

139 >>> vocab = [12, 36, 1138, 42] 

140 >>> data = tf.constant([[12, 1138, 42], [42, 1000, 36]]) # Note OOV tokens 

141 >>> layer = tf.keras.layers.IntegerLookup(vocabulary=vocab) 

142 >>> layer(data) 

143 <tf.Tensor: shape=(2, 3), dtype=int64, numpy= 

144 array([[1, 3, 4], 

145 [4, 0, 2]])> 

146 

147 **Creating a lookup layer with an adapted vocabulary** 

148 

149 This example creates a lookup layer and generates the vocabulary by 

150 analyzing the dataset. 

151 

152 >>> data = tf.constant([[12, 1138, 42], [42, 1000, 36]]) 

153 >>> layer = tf.keras.layers.IntegerLookup() 

154 >>> layer.adapt(data) 

155 >>> layer.get_vocabulary() 

156 [-1, 42, 1138, 1000, 36, 12] 

157 

158 Note that the OOV token -1 have been added to the vocabulary. The remaining 

159 tokens are sorted by frequency (42, which has 2 occurrences, is first) then 

160 by inverse sort order. 

161 

162 >>> data = tf.constant([[12, 1138, 42], [42, 1000, 36]]) 

163 >>> layer = tf.keras.layers.IntegerLookup() 

164 >>> layer.adapt(data) 

165 >>> layer(data) 

166 <tf.Tensor: shape=(2, 3), dtype=int64, numpy= 

167 array([[5, 2, 1], 

168 [1, 3, 4]])> 

169 

170 

171 **Lookups with multiple OOV indices** 

172 

173 This example demonstrates how to use a lookup layer with multiple OOV 

174 indices. When a layer is created with more than one OOV index, any OOV 

175 tokens are hashed into the number of OOV buckets, distributing OOV tokens in 

176 a deterministic fashion across the set. 

177 

178 >>> vocab = [12, 36, 1138, 42] 

179 >>> data = tf.constant([[12, 1138, 42], [37, 1000, 36]]) 

180 >>> layer = tf.keras.layers.IntegerLookup( 

181 ... vocabulary=vocab, num_oov_indices=2) 

182 >>> layer(data) 

183 <tf.Tensor: shape=(2, 3), dtype=int64, numpy= 

184 array([[2, 4, 5], 

185 [1, 0, 3]])> 

186 

187 Note that the output for OOV token 37 is 1, while the output for OOV token 

188 1000 is 0. The in-vocab terms have their output index increased by 1 from 

189 earlier examples (12 maps to 2, etc) in order to make space for the extra 

190 OOV token. 

191 

192 **One-hot output** 

193 

194 Configure the layer with `output_mode='one_hot'`. Note that the first 

195 `num_oov_indices` dimensions in the ont_hot encoding represent OOV values. 

196 

197 >>> vocab = [12, 36, 1138, 42] 

198 >>> data = tf.constant([12, 36, 1138, 42, 7]) # Note OOV tokens 

199 >>> layer = tf.keras.layers.IntegerLookup( 

200 ... vocabulary=vocab, output_mode='one_hot') 

201 >>> layer(data) 

202 <tf.Tensor: shape=(5, 5), dtype=float32, numpy= 

203 array([[0., 1., 0., 0., 0.], 

204 [0., 0., 1., 0., 0.], 

205 [0., 0., 0., 1., 0.], 

206 [0., 0., 0., 0., 1.], 

207 [1., 0., 0., 0., 0.]], dtype=float32)> 

208 

209 **Multi-hot output** 

210 

211 Configure the layer with `output_mode='multi_hot'`. Note that the first 

212 `num_oov_indices` dimensions in the multi_hot encoding represent OOV tokens 

213 

214 >>> vocab = [12, 36, 1138, 42] 

215 >>> data = tf.constant([[12, 1138, 42, 42], 

216 ... [42, 7, 36, 7]]) # Note OOV tokens 

217 >>> layer = tf.keras.layers.IntegerLookup( 

218 ... vocabulary=vocab, output_mode='multi_hot') 

219 >>> layer(data) 

220 <tf.Tensor: shape=(2, 5), dtype=float32, numpy= 

221 array([[0., 1., 0., 1., 1.], 

222 [1., 0., 1., 0., 1.]], dtype=float32)> 

223 

224 **Token count output** 

225 

226 Configure the layer with `output_mode='count'`. As with multi_hot output, 

227 the first `num_oov_indices` dimensions in the output represent OOV tokens. 

228 

229 >>> vocab = [12, 36, 1138, 42] 

230 >>> data = tf.constant([[12, 1138, 42, 42], 

231 ... [42, 7, 36, 7]]) # Note OOV tokens 

232 >>> layer = tf.keras.layers.IntegerLookup( 

233 ... vocabulary=vocab, output_mode='count') 

234 >>> layer(data) 

235 <tf.Tensor: shape=(2, 5), dtype=float32, numpy= 

236 array([[0., 1., 0., 1., 2.], 

237 [2., 0., 1., 0., 1.]], dtype=float32)> 

238 

239 **TF-IDF output** 

240 

241 Configure the layer with `output_mode='tf_idf'`. As with multi_hot output, 

242 the first `num_oov_indices` dimensions in the output represent OOV tokens. 

243 

244 Each token bin will output `token_count * idf_weight`, where the idf weights 

245 are the inverse document frequency weights per token. These should be 

246 provided along with the vocabulary. Note that the `idf_weight` for OOV 

247 tokens will default to the average of all idf weights passed in. 

248 

249 >>> vocab = [12, 36, 1138, 42] 

250 >>> idf_weights = [0.25, 0.75, 0.6, 0.4] 

251 >>> data = tf.constant([[12, 1138, 42, 42], 

252 ... [42, 7, 36, 7]]) # Note OOV tokens 

253 >>> layer = tf.keras.layers.IntegerLookup( 

254 ... output_mode='tf_idf', vocabulary=vocab, idf_weights=idf_weights) 

255 >>> layer(data) 

256 <tf.Tensor: shape=(2, 5), dtype=float32, numpy= 

257 array([[0. , 0.25, 0. , 0.6 , 0.8 ], 

258 [1.0 , 0. , 0.75, 0. , 0.4 ]], dtype=float32)> 

259 

260 To specify the idf weights for oov tokens, you will need to pass the entire 

261 vocabularly including the leading oov token. 

262 

263 >>> vocab = [-1, 12, 36, 1138, 42] 

264 >>> idf_weights = [0.9, 0.25, 0.75, 0.6, 0.4] 

265 >>> data = tf.constant([[12, 1138, 42, 42], 

266 ... [42, 7, 36, 7]]) # Note OOV tokens 

267 >>> layer = tf.keras.layers.IntegerLookup( 

268 ... output_mode='tf_idf', vocabulary=vocab, idf_weights=idf_weights) 

269 >>> layer(data) 

270 <tf.Tensor: shape=(2, 5), dtype=float32, numpy= 

271 array([[0. , 0.25, 0. , 0.6 , 0.8 ], 

272 [1.8 , 0. , 0.75, 0. , 0.4 ]], dtype=float32)> 

273 

274 When adapting the layer in tf_idf mode, each input sample will be considered 

275 a document, and idf weight per token will be calculated as 

276 `log(1 + num_documents / (1 + token_document_count))`. 

277 

278 **Inverse lookup** 

279 

280 This example demonstrates how to map indices to tokens using this layer. 

281 (You can also use `adapt()` with `inverse=True`, but for simplicity we'll 

282 pass the vocab in this example.) 

283 

284 >>> vocab = [12, 36, 1138, 42] 

285 >>> data = tf.constant([[1, 3, 4], [4, 0, 2]]) 

286 >>> layer = tf.keras.layers.IntegerLookup(vocabulary=vocab, invert=True) 

287 >>> layer(data) 

288 <tf.Tensor: shape=(2, 3), dtype=int64, numpy= 

289 array([[ 12, 1138, 42], 

290 [ 42, -1, 36]])> 

291 

292 Note that the first index correspond to the oov token by default. 

293 

294 

295 **Forward and inverse lookup pairs** 

296 

297 This example demonstrates how to use the vocabulary of a standard lookup 

298 layer to create an inverse lookup layer. 

299 

300 >>> vocab = [12, 36, 1138, 42] 

301 >>> data = tf.constant([[12, 1138, 42], [42, 1000, 36]]) 

302 >>> layer = tf.keras.layers.IntegerLookup(vocabulary=vocab) 

303 >>> i_layer = tf.keras.layers.IntegerLookup( 

304 ... vocabulary=layer.get_vocabulary(), invert=True) 

305 >>> int_data = layer(data) 

306 >>> i_layer(int_data) 

307 <tf.Tensor: shape=(2, 3), dtype=int64, numpy= 

308 array([[ 12, 1138, 42], 

309 [ 42, -1, 36]])> 

310 

311 In this example, the input token 1000 resulted in an output of -1, since 

312 1000 was not in the vocabulary - it got represented as an OOV, and all OOV 

313 tokens are returned as -1 in the inverse layer. Also, note that for the 

314 inverse to work, you must have already set the forward layer vocabulary 

315 either directly or via `adapt()` before calling `get_vocabulary()`. 

316 """ 

317 

318 def __init__( 

319 self, 

320 max_tokens=None, 

321 num_oov_indices=1, 

322 mask_token=None, 

323 oov_token=-1, 

324 vocabulary=None, 

325 vocabulary_dtype="int64", 

326 idf_weights=None, 

327 invert=False, 

328 output_mode="int", 

329 sparse=False, 

330 pad_to_max_tokens=False, 

331 **kwargs, 

332 ): 

333 if not tf.dtypes.as_dtype(vocabulary_dtype).is_integer: 

334 raise ValueError( 

335 "`vocabulary_dtype` must be an integer dtype. " 

336 f"Received: {vocabulary_dtype}" 

337 ) 

338 

339 # Legacy versions of the IntegerLookup layer set layer dtype to int64, 

340 # instead of the output type. If we see this and output mode is not 

341 # "int", clear the setting so we don't switch types for old SavedModels. 

342 if ( 

343 output_mode != "int" 

344 and "dtype" in kwargs 

345 and (kwargs["dtype"] == tf.int64 or kwargs["dtype"] == "int64") 

346 ): 

347 del kwargs["dtype"] 

348 

349 # Support deprecated args for this layer. 

350 if "max_values" in kwargs: 

351 logging.log_first_n( 

352 logging.WARN, 

353 "max_values is deprecated, use max_tokens instead.", 

354 1, 

355 ) 

356 max_tokens = kwargs["max_values"] 

357 del kwargs["max_values"] 

358 if "mask_value" in kwargs: 

359 logging.log_first_n( 

360 logging.WARN, 

361 "mask_value is deprecated, use mask_token instead.", 

362 1, 

363 ) 

364 mask_token = kwargs["mask_value"] 

365 del kwargs["mask_value"] 

366 if "oov_value" in kwargs: 

367 logging.log_first_n( 

368 logging.WARN, 

369 "oov_value is deprecated, use oov_token instead.", 

370 1, 

371 ) 

372 oov_token = kwargs["oov_value"] 

373 del kwargs["oov_value"] 

374 

375 # If max_tokens is set, the token must be greater than 1 - otherwise we 

376 # are creating a 0-element vocab, which doesn't make sense. 

377 if max_tokens is not None and max_tokens <= 1: 

378 raise ValueError( 

379 "If `max_tokens` is set for `IntegerLookup`, it must be " 

380 f"greater than 1. Received: max_tokens={max_tokens}." 

381 ) 

382 

383 if num_oov_indices < 0: 

384 raise ValueError( 

385 "The value of `num_oov_indices` argument for `IntegerLookup` " 

386 "must >= 0. Received num_oov_indices=" 

387 f"{num_oov_indices}." 

388 ) 

389 

390 # Make sure mask and oov are of the dtype we want. 

391 mask_token = None if mask_token is None else np.int64(mask_token) 

392 oov_token = None if oov_token is None else np.int64(oov_token) 

393 

394 super().__init__( 

395 max_tokens=max_tokens, 

396 num_oov_indices=num_oov_indices, 

397 mask_token=mask_token, 

398 oov_token=oov_token, 

399 vocabulary=vocabulary, 

400 vocabulary_dtype=vocabulary_dtype, 

401 idf_weights=idf_weights, 

402 invert=invert, 

403 output_mode=output_mode, 

404 sparse=sparse, 

405 pad_to_max_tokens=pad_to_max_tokens, 

406 **kwargs, 

407 ) 

408 base_preprocessing_layer.keras_kpl_gauge.get_cell("IntegerLookup").set( 

409 True 

410 ) 

411 

412 # We override this method solely to generate a docstring. 

413 def adapt(self, data, batch_size=None, steps=None): 

414 """Computes a vocabulary of interger terms from tokens in a dataset. 

415 

416 Calling `adapt()` on an `IntegerLookup` layer is an alternative to 

417 passing in a precomputed vocabulary on construction via the 

418 `vocabulary` argument. An `IntegerLookup` layer should always be either 

419 adapted over a dataset or supplied with a vocabulary. 

420 

421 During `adapt()`, the layer will build a vocabulary of all integer 

422 tokens seen in the dataset, sorted by occurrence count, with ties broken 

423 by sort order of the tokens (high to low). At the end of `adapt()`, if 

424 `max_tokens` is set, the vocabulary wil be truncated to `max_tokens` 

425 size. For example, adapting a layer with `max_tokens=1000` will compute 

426 the 1000 most frequent tokens occurring in the input dataset. If 

427 `output_mode='tf-idf'`, `adapt()` will also learn the document 

428 frequencies of each token in the input dataset. 

429 

430 In order to make `StringLookup` efficient in any distribution context, 

431 the vocabulary is kept static with respect to any compiled `tf.Graph`s 

432 that call the layer. As a consequence, if the layer is adapted a second 

433 time, any models using the layer should be re-compiled. For more 

434 information see 

435 `tf.keras.layers.experimental.preprocessing.PreprocessingLayer.adapt`. 

436 

437 `adapt()` is meant only as a single machine utility to compute layer 

438 state. To analyze a dataset that cannot fit on a single machine, see 

439 [Tensorflow Transform]( 

440 https://www.tensorflow.org/tfx/transform/get_started) for a 

441 multi-machine, map-reduce solution. 

442 

443 Arguments: 

444 data: The data to train on. It can be passed either as a 

445 `tf.data.Dataset`, or as a numpy array. 

446 batch_size: Integer or `None`. 

447 Number of samples per state update. 

448 If unspecified, `batch_size` will default to 32. 

449 Do not specify the `batch_size` if your data is in the 

450 form of datasets, generators, or `keras.utils.Sequence` instances 

451 (since they generate batches). 

452 steps: Integer or `None`. 

453 Total number of steps (batches of samples) 

454 When training with input tensors such as 

455 TensorFlow data tensors, the default `None` is equal to 

456 the number of samples in your dataset divided by 

457 the batch size, or 1 if that cannot be determined. If x is a 

458 `tf.data` dataset, and 'steps' is None, the epoch will run until 

459 the input dataset is exhausted. When passing an infinitely 

460 repeating dataset, you must specify the `steps` argument. This 

461 argument is not supported with array inputs. 

462 """ 

463 super().adapt(data, batch_size=batch_size, steps=steps) 

464