Get your own copy of wikidata with qlever

[WolfgangFahl]

Currently i am busy with my main project so it might take another 2-3 weeks before i can continue more in-depth experiments for https://wiki.bitplan.com/index.php/Get_your_own_copy_of_WikiData

Since https://wiki.bitplan.com/index.php/WikiData_Import_2022-01-29 was a success
and
the two attempts
https://wiki.bitplan.com/index.php/WikiData_Import_2022-03-11
and
https://wiki.bitplan.com/index.php/WikiData_Import_2022-03-16

were not the main problem at this point is that the test environment is unstable. The qlever script #562 i originally used was declared as not actual any more by now and if i'd simply retry with it that would IMHO not be considered a valid attempt by you.

For the next run i intend to try a smaller dataset for simply trying out the setup and then do the wikidata import. That way general issues will be caught much earlier on. Ideally the smaller dataset would be a useful subset of wikidata which i might be able to generate with my Trulytabular tool by only selecting relevant item classes and properties.

• academic conference
• scientific conference series
• proceedings

for the "bigger" classes "author" and "paper" qlever is most interesting for me since the WikdataQueryService times out on the truly tabular analysis attempts.

[WolfgangFahl]

The latest attempt is documented at https://wiki.bitplan.com/index.php/WikiData_Import_2022-05-21.
This attempt failed see ad-freiburg/qlever-control#4

[WolfgangFahl]

https://wiki.bitplan.com/index.php/WikiData_Import_2022-05-22 is started.

ad-freiburg/qlever-control#4 hit again - retrying another time ...

[WolfgangFahl]

I could do the next wikidata import trial on multiple machines:

1. the 128 GB Ubuntu Dell used server from 2022 (some 10 years old)
2. the 64 GB Ubunutu used server from 2018 (some 14 years old)
3. the 64 GB Mac OS server (some 10 years old)
4. an Amazon Web cloud based machine
5. the new server to be installed soon at my department

Options 1-3 are readily available and used to work in 2018 and i see no reason why they shouldn't still work since the machine just run the OS and dozens of other software packages fine. As for the instructions i would git clone/pull the qlever-control script and qlever c++ software - set up the packages as outlined in https://wiki.bitplan.com/index.php/WikiData_Import_2022-03-16#Native_approach and recompile and then run the qlever-control script

For me the biggest issue for reproducability is still https://wiki.bitplan.com/index.php/WikiData_Import_2022-03-16#Native_approach because the libaries of the O/S seem the need to be tweaked. If we can't use docker to make sure these prerequisites are met i think this should also be scripted and checked otherwise the reproducability might suffer. is there any chance for my mac O/S environment?

[joka921]

Hi Wolfgang,

The first step we should do is find out, what went wrong with your last attempt. I repost my question which needs to be answered to help you:

The problem in your concrete case seems to be that the wikidata.settings.json seems to have the wrong contents. The "official" one sets appropriate externalization and also a default locale (the locale is not super important, but it helped me confim my theory).
Can you send us the wikidata.settings.json you used?

Concerning the more general "Reproducible binaries besides Docker" we have several thoughts since we recently shifted our mindset from "Use linux use docker" to "more people seem to be interested in native QLever on Mac". We are still figuring out the best way to tackle those problems (we want something that runs natively because of performance issues, but we don't want the dependencies to be encapsulated and not spill into the global system configuration. One Idea which seems to be feasible and was suggested (we have run initial tests, but there's quite some work to do), seems to be the cross-compilation toolchain of the Julia project.

[WolfgangFahl]

@joka921 see https://wiki.bitplan.com/index.php/WikiData_Import_2022-05-22#wikidata.settings.json - it only sets the numtriples per batch

[WolfgangFahl]

https://wiki.bitplan.com/index.php/WikiData_Import_2022-06-24 has the report for the latest attempt

[WolfgangFahl]

https://wiki.bitplan.com/index.php/WikiData_Import_2022-06-25 has the report for the latest attempt.
This attempt is with 97% disk space on the the 4 TB disk and freshly downloaded wikidata data with 17.5 billion triples

[hannahbast]

Are you saying it worked 🙂

Can you post the output of qlever index-stats

[WolfgangFahl]

@hannahbast -

looks like the IndexBuilder is running.

the last few lines of the log are:
2022-06-27 04:05:48.950 - INFO: Creating a pair of index permutations ...
2022-06-27 06:00:17.967 - INFO: Statistics for SPO: #relations = 2,808,258,381, #blocks = 30,045, #triples = 23,623,846,556
2022-06-27 06:00:17.967 - INFO: Statistics for SOP: #relations = 2,808,258,381, #blocks = 30,045, #triples = 23,623,846,556
2022-06-27 06:00:17.967 - INFO: Exchanging multiplicities for SPO and SOP ...
2022-06-27 06:42:31.181 - INFO: Writing meta data for SPO and SOP ...
2022-06-27 06:42:47.139 - INFO: Number of distinct patterns: 7,168,422
2022-06-27 06:42:47.143 - INFO: Number of subjects with pattern: 2,808,258,381 [all]
2022-06-27 06:42:47.143 - INFO: Total number of distinct subject-predicate pairs: 11,082,607,100
2022-06-27 06:42:47.143 - INFO: Average number of predicates per subject: 3.9
2022-06-27 06:42:47.145 - INFO: Average number of subjects per predicate: 236,545

I don't know what the expected output of the log is with this version - my logstat scripts assumes entries as they were found in the february log. In the last few crashes there was no core dump or logentry so there is no graceful handling of out of memory or out of disk space situations - each of these should IMHO be separate tickets.

I also would appreciate if the logfile would have a "I am still alive" entry after a specified time say e.g. 1h so that it is not necessary to look for the process state separately when monitoring the indexing see #565.

The last successful attempt on this machine took some 4 days - after the recent speedups things might go a bit quicker - let's see what the timing will be for this attempt.

[WolfgangFahl]

In the meantime the process seems to have stopped and the result is:

ls -l
total 962223076
-rw-rw-r-- 1 wf wf  95427655992 Jun 23 10:50 latest-all.ttl.bz2
-rw-rw-r-- 1 wf wf    329141448 Jun 25 01:28 latest-lexemes.ttl.bz2
-rw-rw-r-- 1 wf wf         1188 Jun 26 08:08 Qleverfile
drwxrwxr-x 2 wf wf         4096 Jun 24 10:09 RCS
-rw-rw-r-- 1 wf wf        38295 Jun 27 10:39 wikidata.index-log.txt
-rw-rw-r-- 1 wf wf  36326088459 Jun 27 10:39 wikidata.index.ops
-rw-rw-r-- 1 wf wf 110210977824 Jun 27 10:39 wikidata.index.ops.meta
-rw-rw-r-- 1 wf wf  35577144928 Jun 27 10:39 wikidata.index.osp
-rw-rw-r-- 1 wf wf 110210977824 Jun 27 10:39 wikidata.index.osp.meta
-rw-rw-r-- 1 wf wf  15204218196 Jun 27 06:42 wikidata.index.patterns
-rw-rw-r-- 1 wf wf  56975017003 Jun 27 02:41 wikidata.index.pos
-rw-rw-r-- 1 wf wf  65856748771 Jun 27 02:41 wikidata.index.pso
-rw-rw-r-- 1 wf wf  36697218365 Jun 27 06:42 wikidata.index.sop
-rw-rw-r-- 1 wf wf 110210977824 Jun 27 06:42 wikidata.index.sop.meta
-rw-rw-r-- 1 wf wf  36954646077 Jun 27 06:42 wikidata.index.spo
-rw-rw-r-- 1 wf wf 110210977824 Jun 27 06:42 wikidata.index.spo.meta
-rw-rw-r-- 1 wf wf          302 Jun 27 10:39 wikidata.meta-data.json
-rw-rw-r-- 1 wf wf         4399 Jun 26 20:36 wikidata.prefixes
-rw-rw-r-- 1 wf wf          362 Jun 26 08:14 wikidata.settings.json
-rw-rw-r-- 1 wf wf 156706000672 Jun 26 18:11 wikidata.vocabulary.external
-rw-rw-r-- 1 wf wf  48982650912 Jun 26 18:11 wikidata.vocabulary.external.idsAndOffsets.mmap
-rw-rw-r-- 1 wf wf  27559938325 Jun 26 20:46 wikidata.vocabulary.internal

is that the expected result?

The logfile seems unchanged and there is no success or finish message. I fear this might be because i had to delete some lines from the nohup.out result while the process is running.

[joka921]

1. A small hint: If you use ls -lh instead of ls -l aka ll then the sizes of the files will be shown in a human-readable format (110 GB etc). That makes it easier to parse.
2. All files that should be there are there and have roughly the expected size.
3. There should be a "finished successfully" message, but you stated the possibility that this was lost in your pipeline. For that reason we tee and pipe the log in the qlever script, so the wikidata.index-log.txt file should contain the full log, please check how that log ends.

[hannahbast]

@WolfgangFahl I would strongly suggest to exactly follow the instructions on https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-Wikidata . It's maximally easy and I have done it many times on many different machines already and it always worked like a charm (in around 14 hours on our machines). Make sure that you use the latest version of the qlever script!

[WolfgangFahl]

I think we are fine this time ...🎆

tail wikidata.index-log.txt 
2022-06-27 06:42:47.143	- INFO:  Number of subjects with pattern: 2,808,258,381 [all]
2022-06-27 06:42:47.143	- INFO:  Total number of distinct subject-predicate pairs: 11,082,607,100
2022-06-27 06:42:47.143	- INFO:  Average number of predicates per subject: 3.9
2022-06-27 06:42:47.145	- INFO:  Average number of subjects per predicate: 236,545
2022-06-27 08:07:24.260	- INFO:  Creating a pair of index permutations ... 
2022-06-27 09:52:23.085	- INFO:  Statistics for OSP: #relations = 3,179,186,092, #blocks = 39,688, #triples = 23,623,846,556
2022-06-27 09:52:23.101	- INFO:  Statistics for OPS: #relations = 3,179,186,092, #blocks = 39,688, #triples = 23,623,846,556
2022-06-27 09:52:23.101	- INFO:  Exchanging multiplicities for OSP and OPS ...
2022-06-27 10:39:15.273	- INFO:  Writing meta data for OSP and OPS ...
2022-06-27 10:39:20.510	- INFO:  Index build completed

#565 has the logstats script i am using now - the exected values for the phases are needed as input:

expectedTriples=17500 -v expectedBoM=800 -v expectedUoM=3200 -v expectedConversion=28300 -v expectedWords=800

The next step will be to restart the server and retry the frontend

[WolfgangFahl]

@hannahbast could you please give me a hint for your 14h run to be included in https://wiki.bitplan.com/index.php/Get_your_own_copy_of_WikiData#Performance_Reports

[hannahbast]

@WolfgangFahl What exactly is your question? Have you read https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-Wikidata ? Have you looked at the QLeverfile produced by the qlever script for Wikidata? It contains all the relevant settings.

[hannahbast]

I just looked at https://wiki.bitplan.com/index.php/Get_your_own_copy_of_WikiData#Performance_Reports . I don't think it makes a lot of sense to have a table with absolute running times, but on totally different machines. As a minimum, the table should also include the processor specification, the number of cores and the amount of RAM used.

And since you include unverified information from https://lists.wikimedia.org/hyperkitty/list/[email protected]/message/CKN3QPV2NJ5TAKHORMYDTTXG7Y65UXAF , you should also include a line with the 14 hours from https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-Wikidata .

[WolfgangFahl]

@hannahbast thank you for the feedback - i have updated the table to include more information

[hannahbast]

By the way: Thanks for including Stardog in your evaluation. I just tried it myself for the DBLP dataset (266M triples), which is large enough to be interesting, but small enough so that one play around with it easily. Here are some very preliminary findings:

1. Stardog's loading time is just as fast as that of QLever (9 minutes for DBLP). That is impressive for code written in Java. However, it should be noted that QLever builds several non-trivial index data structures while loading.

2. Stardog's query times are less stellar. I tried the example queries from https://qlever.cs.uni-freiburg.de/dblp . The GROUP BY queries were two orders of magnitude slower with Stardog. When asking the queries again, they were only one order of magnitude slower. So there seems to be some caching going some (would be good to no more about this).

3. The FILTER CONTAINS query was very slow with Stardog (over a minute before caching, 20 seconds when repeating the query vs. 70ms on QLever). Does Stardog have an option for faster text search?

[WolfgangFahl]

we'd love to start our qlever wikidata instance and can't get it work with qlever-control nor with a command line.
we try to follow https://github.com/ad-freiburg/qlever/blob/master/docs/native_setup.md

2022-07-26 10:05:33.372	- INFO:  QLever Server, compiled on Jun 24 2022 09:48:25
2022-07-26 10:05:33.378	- INFO:  Initializing server ...
2022-07-26 10:05:33.379	- ERROR: ASSERT FAILED (f.is_open(); in ../src/index/Index.cpp, line 859, function void Index::readConfiguration())

[hannahbast]

1. We haven't update the native_setup.md in a while and it's really not needed anymore
2. The ASSERT failure is a hint that you are not using the latest version of Qlever; see Proper error message when a file can't be opened #731
3. You should use the Qleverfile produced by the qlever script with ". qlever wikidata"
4. If you post your Qleverfile and the list of files in your directory, I can tell you what went wrong

[WolfgangFahl]

@hannahbast @joka921 - thanks for helping with this. https://wiki.bitplan.com/index.php/WikiData_Import_2022-06-24#Starting_server has the description of the working start command. I still have trouble with the GUI but the server itself works.