{
  "citation": "@article{mehta2021alx,\n    title={ALX: Large Scale Matrix Factorization on TPUs},\n    author={Harsh Mehta and Steffen Rendle and Walid Krichene and Li Zhang},\n    year={2021},\n    eprint={2112.02194},\n    archivePrefix={arXiv},\n    primaryClass={cs.LG}\n}",
  "configDescription": "WebGraph-de-dense contains around 0.82B edges and around 5.7M\nnodes.\n",
  "configName": "de-dense",
  "description": "This dataset contains a sparse graph representing web link structure for a\nsmall subset of the Web.\n\nIts a processed version of a single crawl performed by CommonCrawl in 2021\nwhere we strip everything and keep only the link->outlinks structure.\nThe final dataset is basically int -> List[int] format with each integer id\nrepresenting a url.\n\nAlso, in order to increase the value of this resource, we created 6 different\nversion of WebGraph, each varying in the sparsity pattern and locale. We took\nthe following processing steps, in order:\n\n- We started with WAT files from June 2021 crawl.\n- Since the outlinks in HTTP-Response-Metadata are stored as relative paths, we\nconvert them to absolute paths using urllib after validating each link.\n- To study locale-specific graphs, we further filter based on 2 top level\ndomains: \u2018de\u2019 and \u2018in\u2019, each producing a graph with an order of magnitude less\nnumber of nodes.\n- These graphs can still have arbitrary sparsity patterns and dangling links.\nThus we further filter the nodes in each graph to have minimum of K \u2208 [10, 50]\ninlinks and outlinks. Note that we only do this processing once, thus this is\nstill an approximation i.e. the resulting graph might have nodes with less than\nK links.\n- Using both locale and count filters, we finalize 6 versions of WebGraph\ndataset, summarized in the folling table.\n\n| Version      | Top level domain | Min count | Num nodes | Num edges |\n| -----------  | ---------------- | --------- | --------- | --------- |\n| sparse       |                  | 10        | 365.4M    | 30B       |\n| dense        |                  | 50        | 136.5M    | 22B       |\n| de-sparse    | de               | 10        | 19.7M     | 1.19B     |\n| de-dense     | de               | 50        | 5.7M      | 0.82B     |\n| in-sparse    | in               | 10        | 1.5M      | 0.14B     |\n| in-dense     | in               | 50        | 0.5M      | 0.12B     |\n\nAll versions of the dataset have following features:\n\n- \"row_tag\": a unique identifier of the row (source link).\n- \"col_tag\": a list of unique identifiers of non-zero columns (dest outlinks).\n- \"gt_tag\": a list of unique identifiers of non-zero columns used as ground\ntruth (dest outlinks), empty for train/train_t splits.",
  "fileFormat": "tfrecord",
  "location": {
    "urls": [
      "https://arxiv.org/abs/2112.02194"
    ]
  },
  "moduleName": "tensorflow_datasets.structured.web_graph.web_graph",
  "name": "web_graph",
  "releaseNotes": {
    "1.0.0": "Initial release."
  },
  "splits": [
    {
      "filepathTemplate": "{DATASET}-{SPLIT}.{FILEFORMAT}-{SHARD_X_OF_Y}",
      "name": "train",
      "numBytes": "2987505824",
      "shardLengths": [
        "159966",
        "159965",
        "159966",
        "159966",
        "159965",
        "159966",
        "159966",
        "159966",
        "159965",
        "159966",
        "159966",
        "159965",
        "159966",
        "159966",
        "159965",
        "159966",
        "159966",
        "159965",
        "159966",
        "159966",
        "159965",
        "159966",
        "159966",
        "159965",
        "159966",
        "159966",
        "159966",
        "159965",
        "159966",
        "159966",
        "159965",
        "159966"
      ]
    },
    {
      "filepathTemplate": "{DATASET}-{SPLIT}.{FILEFORMAT}-{SHARD_X_OF_Y}",
      "name": "train_t",
      "numBytes": "3018328458",
      "shardLengths": [
        "177265",
        "177265",
        "177264",
        "177265",
        "177265",
        "177265",
        "177264",
        "177265",
        "177265",
        "177265",
        "177265",
        "177264",
        "177265",
        "177265",
        "177265",
        "177264",
        "177265",
        "177265",
        "177265",
        "177265",
        "177264",
        "177265",
        "177265",
        "177265",
        "177265",
        "177264",
        "177265",
        "177265",
        "177265",
        "177264",
        "177265",
        "177265"
      ]
    },
    {
      "filepathTemplate": "{DATASET}-{SPLIT}.{FILEFORMAT}-{SHARD_X_OF_Y}",
      "name": "test",
      "numBytes": "330782850",
      "shardLengths": [
        "138318",
        "138317",
        "138317",
        "138318"
      ]
    }
  ],
  "version": "1.0.0"
}