Clustering

Clustering

Cluster documents into clusters

import os.path import pandas as pd from time import time import requests pd.options.display.float_format = '{:,.3f}'.format dataset_name = "treclegal09_2k_subset" # see list of available datasets BASE_URL = "http://localhost:5001/api/v0" # FreeDiscovery server URL

0. Load the example dataset

url = BASE_URL + '/example-dataset/{}'.format(dataset_name) print(" GET", url) input_ds = requests.get(url).json() # To use a custom dataset, simply specify the following variables data_dir = input_ds['metadata']['data_dir'] dataset_definition = [{'document_id': row['document_id'], 'file_path': os.path.join(data_dir, row['file_path'])} for row in input_ds['dataset']]

Out:

GET http://localhost:5001/api/v0/example-dataset/treclegal09_2k_subset

1. Feature extraction (non hashed)

1.a Load dataset and initalize feature extraction

url = BASE_URL + '/feature-extraction' print(" POST", url) res = requests.post(url).json() dsid = res['id'] print(" => received {}".format(list(res.keys()))) print(" => dsid = {}".format(dsid))

Out:

POST http://localhost:5001/api/v0/feature-extraction
   => received ['id']
   => dsid = 6def156805734009

1.b Run feature extraction

url = BASE_URL+'/feature-extraction/{}'.format(dsid) print(" POST", url) res = requests.post(url, json={'dataset_definition': dataset_definition})

Out:

POST http://localhost:5001/api/v0/feature-extraction/6def156805734009

2. Calculate LSI

url = BASE_URL + '/lsi/' print("POST", url) n_components = 300 res = requests.post(url, json={'n_components': n_components, 'parent_id': dsid }).json() lsi_id = res['id'] print(' => LSI model id = {}'.format(lsi_id)) print((' => SVD decomposition with {} dimensions ' 'explaining {:.2f} % variabilty of the data') .format(n_components, res['explained_variance']*100))

Out:

POST http://localhost:5001/api/v0/lsi/
  => LSI model id = 06abfedddb1e4dfe
  => SVD decomposition with 300 dimensions explaining 87.16 % variabilty of the data

3. Document Clustering (LSI + K-Means)

print("\n3.a. Document clustering (LSI + K-means)") url = BASE_URL + '/clustering/k-mean/' print(" POST", url) t0 = time() res = requests.post(url, json={'parent_id': lsi_id, 'n_clusters': 10, }).json() mid = res['id'] print(" => model id = {}".format(mid))

Out:

3.a. Document clustering (LSI + K-means)
 POST http://localhost:5001/api/v0/clustering/k-mean/
     => model id = 559f3f5ff3dd41e4

3.b. Computing cluster labels

url = BASE_URL + '/clustering/k-mean/{}'.format(mid) print(" GET", url) res = requests.get(url, json={'n_top_words': 3 }).json() t1 = time() data = res['data'] for row in data: row['n_documents'] = len(row.pop('documents')) print(' .. computed in {:.1f}s'.format(t1 - t0)) print(pd.DataFrame(data))

Out:

GET http://localhost:5001/api/v0/clustering/k-mean/559f3f5ff3dd41e4
    .. computed in 1.2s
   cluster_id                   cluster_label  cluster_similarity  cluster_size  n_documents
0           0                   doc gtc shall               0.269           114          114
1           1           conference mon normal               0.197           250          250
2           2          migration outlook team               0.309           148          148
3           3          rewrite address server               1.000            56           56
4           4                tana jones enron               0.192           200          200
5           5                nov tenet normal               0.223           258          258
6           6              tue shackleton oct               0.284           136          136
7           7                  ect hou neuner               0.414           102          102
8           8              teneo test subject               0.080          1012         1012
9           9  test recipients administrative               0.247           189          189

4. Document Clustering (LSI + Birch Clustering)

print("\n4.a. Document clustering (LSI + Birch clustering)") url = BASE_URL + '/clustering/birch/' print(" POST", url) t0 = time() res = requests.post(url, json={'parent_id': lsi_id, 'n_clusters': -1, 'min_similarity': 0.7, 'branching_factor': 20, 'max_tree_depth': 2, }).json() mid = res['id'] print(" => model id = {}".format(mid))

Out:

4.a. Document clustering (LSI + Birch clustering)
 POST http://localhost:5001/api/v0/clustering/birch/
     => model id = f4897ee343ca437d

4.b. Computing cluster labels

url = BASE_URL + '/clustering/birch/{}'.format(mid) print(" GET", url) res = requests.get(url, json={'n_top_words': 3 }).json() t1 = time() print(' .. computed in {:.1f}s'.format(t1 - t0)) data = res['data'] for row in data: row['n_documents'] = len(row.pop('documents')) print(pd.DataFrame(data))

Out:

GET http://localhost:5001/api/v0/clustering/birch/f4897ee343ca437d
    .. computed in 1.9s
                                             children  cluster_depth  cluster_id                        cluster_label  cluster_similarity  cluster_size  n_documents
0   [1, 2, 12, 24, 25, 28, 41, 42, 43, 47, 48, 58,...              0           0                      normal test ect               0.074          2465         2465
1                                                  []              1           1                    deal aquilla muni               0.684            17           17
2                       [3, 4, 5, 6, 7, 8, 9, 10, 11]              1           2       test shackleton administrative               0.187           258          258
3                                                  []              2           3                    rate group public               0.255            36           36
4                                                  []              2           4               shackleton load normal               0.396            47           47
5                                                  []              2           5               calo shackleton dinner               0.673            16           16
6                                                  []              2           6             deseret etringer counsel               0.537            22           22
7                                                  []              2           7             sample financial trading               0.395            18           18
8                                                  []              2           8                      jones tana load               0.404            32           32
9                                                  []              2           9                     nemec doc gallup               0.458            16           16
10                                                 []              2          10           services commission energy               0.391            14           14
11                                                 []              2          11       test recipients administrative               0.446            57           57
12       [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]              1          12                  rewrite address ect               0.219           372          372
13                                                 []              2          13        issues cone enron_development               0.470            26           26
14                                                 []              2          14                       ect hou london               0.380            39           39
15                                                 []              2          15               rewrite address server               0.531           114          114
16                                                 []              2          16                      memo week clair               0.337            24           24
17                                                 []              2          17          attorney confirmation south               0.446            11           11
18                                                 []              2          18                 ena transaction kean               0.394            43           43
19                                                 []              2          19                   tiger bailey sekse               0.487            14           14
20                                                 []              2          20                  test bruno gaillard               0.563            15           15
21                                                 []              2          21                      meet load teneo               0.535            13           13
22                                                 []              2          22                    normal test teneo               0.445            46           46
23                                                 []              2          23                      load teneo test               0.494            27           27
24                                                 []              1          24             amoco jefferson sorenson               0.556            15           15
25                                           [26, 27]              1          25                   sanders normal nov               0.481            63           63
26                                                 []              2          26                   sanders normal nov               0.493            54           54
27                                                 []              2          27               sanders fri conference               0.578             9            9
28   [29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40]              1          28                   normal subject wed               0.120           481          481
29                                                 []              2          29                  lessee lessor nemec               0.337            50           50
..                                                ...            ...         ...                                  ...                 ...           ...          ...
57                                                 []              2          57                        wed nov tenet               0.390            38           38
58                                           [59, 60]              1          58                        tue tenet thu               0.290           107          107
59                                                 []              2          59                     thu tenet normal               0.325            68           68
60                                                 []              2          60                        tue oct tenet               0.471            39           39
61  [62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 7...              1          61       enron_development teneo normal               0.102           687          687
62                                                 []              2          62  enron_development advice catalytica               0.337            77           77
63                                                 []              2          63               ruppert operator exxon               0.355            52           52
64                                                 []              2          64                      teneo test date               0.352            47           47
65                                                 []              2          65            products weather internet               0.387            22           22
66                                                 []              2          66        enron_development gdr hagerty               0.394            32           32
67                                                 []              2          67           trade inflation investment               0.414            18           18
68                                                 []              2          68                   bump doc agreement               0.356            45           45
69                                                 []              2          69                 boyd hunter language               0.366            27           27
70                                                 []              2          70         sampling americancentury rcr               0.320            36           36
71                                                 []              2          71                 houston dickson calo               0.352            29           29
72                                                 []              2          72            attorney fri transactions               0.344            21           21
73                                                 []              2          73                     neuner wed teneo               0.293            32           32
74                                                 []              2          74                  haedicke normal nov               0.413            42           42
75                                                 []              2          75            teneo taylor_mark bockius               0.334            32           32
76                                                 []              2          76              registration legal cftc               0.364            14           14
77                                                 []              2          77                        fri tenet mtg               0.374            26           26
78                                                 []              2          78                     lunch otc energy               0.308            48           48
79                                                 []              2          79             market teneo electricity               0.359            20           20
80                                                 []              2          80                houston shipper nemec               0.444            21           21
81                                                 []              2          81               migration outlook team               0.459            46           46
82                                   [83, 84, 85, 86]              1          82          shackleton_sara teneo group               0.230            82           82
83                                                 []              2          83                        doc isda cini               0.397            33           33
84                                                 []              2          84                  account stay motion               0.395            15           15
85                                                 []              2          85          laryngitis diligence review               0.434            10           10
86                                                 []              2          86          shackleton_sara teneo tiger               0.360            24           24

[87 rows x 7 columns]

5. Optimal sampling (LSI + Birch Clustering)

t0 = time() url = BASE_URL + '/clustering/birch/{}'.format(mid) print(" GET", url) res = requests.get(url, json={'return_optimal_sampling': True, 'sampling_min_coverage': 0.9 }).json() t1 = time() print(' .. computed in {:.1f}s'.format(t1 - t0)) data = res['data'] print(pd.DataFrame(data))

Out:

GET http://localhost:5001/api/v0/clustering/birch/f4897ee343ca437d
    .. computed in 0.1s
    cluster_id  cluster_similarity  cluster_size                                          documents
0           15               0.531           114  [{'document_id': 5041, 'similarity': 0.9245882...
1           62               0.337            77  [{'document_id': 33124, 'similarity': 0.594356...
2           39               0.309            76  [{'document_id': 2735716, 'similarity': 0.5962...
3           59               0.325            68  [{'document_id': 3587236, 'similarity': 0.6634...
4           37               0.325            60  [{'document_id': 2402500, 'similarity': 0.6318...
5           11               0.446            57  [{'document_id': 855625, 'similarity': 0.88858...
6           26               0.493            54  [{'document_id': 3802500, 'similarity': 0.5989...
7           55               0.345            54  [{'document_id': 3171961, 'similarity': 0.6304...
8           63               0.355            52  [{'document_id': 1664100, 'similarity': 0.6798...
9           29               0.337            50  [{'document_id': 714025, 'similarity': 0.62037...
10          33               0.336            50  [{'document_id': 4990756, 'similarity': 0.5743...
11          78               0.308            48  [{'document_id': 1119364, 'similarity': 0.5790...
12           4               0.396            47  [{'document_id': 61504, 'similarity': 0.640083...
13          64               0.352            47  [{'document_id': 2149156, 'similarity': 0.7265...
14          22               0.445            46  [{'document_id': 358801, 'similarity': 0.66319...
15          81               0.459            46  [{'document_id': 5527201, 'similarity': 0.6800...
16          68               0.356            45  [{'document_id': 300304, 'similarity': 0.64720...
17          18               0.394            43  [{'document_id': 79524, 'similarity': 0.603813...
18          74               0.413            42  [{'document_id': 2992900, 'similarity': 0.6323...
19          31               0.296            41  [{'document_id': 2809, 'similarity': 0.4766740...
20          14               0.380            39  [{'document_id': 4422609, 'similarity': 0.7943...
21          60               0.471            39  [{'document_id': 1893376, 'similarity': 0.7655...
22          30               0.366            38  [{'document_id': 627264, 'similarity': 0.73297...
23          57               0.390            38  [{'document_id': 2334784, 'similarity': 0.7788...
24           3               0.255            36  [{'document_id': 1168561, 'similarity': 0.3576...
25          36               0.311            36  [{'document_id': 1232100, 'similarity': 0.4595...
26          70               0.320            36  [{'document_id': 910116, 'similarity': 0.51148...
27          51               0.481            34  [{'document_id': 6041764, 'similarity': 0.7041...
28          32               0.351            33  [{'document_id': 1100401, 'similarity': 0.6933...
29          83               0.397            33  [{'document_id': 4765489, 'similarity': 0.5323...
30           8               0.404            32  [{'document_id': 215296, 'similarity': 0.58934...
31          66               0.394            32  [{'document_id': 91809, 'similarity': 0.806330...
32          73               0.293            32  [{'document_id': 3663396, 'similarity': 0.4763...
33          75               0.334            32  [{'document_id': 3378244, 'similarity': 0.6113...
34          38               0.408            30  [{'document_id': 3359889, 'similarity': 0.6664...
35          44               0.381            30  [{'document_id': 504100, 'similarity': 0.57251...
36          35               0.401            29  [{'document_id': 3164841, 'similarity': 0.5582...
37          71               0.352            29  [{'document_id': 3279721, 'similarity': 0.5513...
38          45               0.348            28  [{'document_id': 1962801, 'similarity': 0.6525...
39          54               0.368            28  [{'document_id': 2468041, 'similarity': 0.5644...
40          23               0.494            27  [{'document_id': 835396, 'similarity': 0.65802...
41          46               0.443            27  [{'document_id': 2076481, 'similarity': 0.8140...
42          69               0.366            27  [{'document_id': 1032256, 'similarity': 0.5915...
43          13               0.470            26  [{'document_id': 289, 'similarity': 0.85353300...
44          77               0.374            26  [{'document_id': 5788836, 'similarity': 0.6740...
45          16               0.337            24  [{'document_id': 16900, 'similarity': 0.457269...
46          50               0.372            24  [{'document_id': 4494400, 'similarity': 0.6840...
47          86               0.360            24  [{'document_id': 4622500, 'similarity': 0.4755...
48          34               0.368            23  [{'document_id': 874225, 'similarity': 0.48114...
49          52               0.384            23  [{'document_id': 2524921, 'similarity': 0.6179...
50           6               0.537            22  [{'document_id': 17689, 'similarity': 0.804906...
51          65               0.387            22  [{'document_id': 1129969, 'similarity': 0.6836...
52          47               0.498            21  [{'document_id': 3118756, 'similarity': 0.7570...
53          49               0.489            21  [{'document_id': 2307361, 'similarity': 0.7566...
54          72               0.344            21  [{'document_id': 4652649, 'similarity': 0.5300...
55          80               0.444            21  [{'document_id': 652864, 'similarity': 0.69850...
56          79               0.359            20  [{'document_id': 142884, 'similarity': 0.59510...
57           7               0.395            18  [{'document_id': 125316, 'similarity': 0.60864...
58          53               0.448            18  [{'document_id': 2669956, 'similarity': 0.7132...
59          67               0.414            18  [{'document_id': 44100, 'similarity': 0.658200...

5. Delete the extracted features

url = BASE_URL + '/feature-extraction/{}'.format(dsid) requests.delete(url)

Total running time of the script: ( 0 minutes 10.439 seconds)