I don't know this is intentional at the moment, but notice this when I was poking around the page.

https://mwmbl.org/ Search for application/opensearchdescription+xml in the mark up. Which shows this:

I think we should be expecting something like this: https://search.brave.com/

<link rel="search" type="application/opensearchdescription+xml" title="Brave Search" href="https://cdn.search.brave.com/serp/v1/static/brand/c57da39655b0b08603d88711f8e33aae50500cbcd8d2fc70a0d01e105cbd0985-opensearch.xml">

Hi everyone,

I'm trying to run a local development env for mwmbl, but I'm experiencing some difficulties.
Been following the dev guide: https://github.com/mwmbl/mwmbl/wiki/Development-FAQ

Steps I've done:

1. Cloning the repo.

git clone [email protected]:raypatterson77/mwmbl.git

3. Setting up Python venv

python -m venv .     

5. Activating Python venv:

 source bin/activate       

7. Install dependencies

 pip install .        

9. Downloaded index File (serveral times) and placed it in data folder
10. Trying to run mwmbl:

mwmbl-tinysearchengine --config config/tinysearchengine.yaml  

Errors:

Running

mwmbl-tinysearchengine --config config/tinysearchengine.yaml  

Gives the error:

usage: mwmbl-tinysearchengine [-h] --index INDEX --terms TERMS
mwmbl-tinysearchengine: error: the following arguments are required: --index, --terms

NOTE: /config/tinysearchengine.yaml is not in the master branch, but still available in the update-readme-for-new-crawler branch. Even with /config/tinysearchengine.yaml present,

mwmbl-tinysearchengine --config config/tinysearchengine.yaml  

is failing with the error above.

Trying to use --index --terms paramters as following:

mwmbl-tinysearchengine --index data/index.tinysearch --terms data/terms.csv  

NOTE: the terms CSV file is nowhere present, not sure about what exactly is expected to be in the file, but from the errors I was getting,

term,count

needs to be present. Creating the file terms.csv under data/terms.csv with above content gives me the error:

Terms [] []
Traceback (most recent call last):
  File "/home/o/Dokumente/code/python/mwmbl-dev/bin/mwmbl-tinysearchengine", line 8, in <module>
    sys.exit(main())
  File "/home/o/Dokumente/code/python/mwmbl-dev/lib/python3.10/site-packages/mwmbl/tinysearchengine/app.py", line 39, in main
    with TinyIndex(item_factory=Document, index_path=args.index) as tiny_index:
  File "/home/o/Dokumente/code/python/mwmbl-dev/lib/python3.10/site-packages/mwmbl/tinysearchengine/indexer.py", line 81, in __init__
    metadata = TinyIndexMetadata.from_bytes(metadata_bytes)
  File "/home/o/Dokumente/code/python/mwmbl-dev/lib/python3.10/site-packages/mwmbl/tinysearchengine/indexer.py", line 51, in from_bytes
    raise ValueError("This doesn't seem to be an index file")
ValueError: This doesn't seem to be an index file

There is no sha sum to validate the integrity of the index.tinysearch file, but as said I downloaded it multiple times and I don't think the file gets corrupted during the download process.

Am I missing something or is there a problem with the resent version of mwmbl? And is there anywhere a example of the terms CSV file? Is config/tinysearchengine.yaml for some reason no longer in the master branch? If not I can make a pull requests to add it back from the update-readme-for-new-crawler branch

Docker

Trying to set up the dev env with docker, ends in similar problems. Unluckily I haven't it documented well, but for building the image config/tinysearchengine.yaml is necessary, as defined in the Dockerfile. After building:

sudo docker run -p 8080:8080 mwmbl                                                                                                                                                                        
usage: mwmbl-tinysearchengine [-h] --index INDEX --terms TERMS
mwmbl-tinysearchengine: error: the following arguments are required: --index, --terms

Env

uname -a                                                                                                                                                                                                   
Linux o 5.16.7-arch1-1

python --version                                                                                                                                                                                          
Python 3.10.2

pip --version                                                                                                                                                                                                 
pip 21.2.4

docker version                                                                                                                                                                                                
Client:
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.17.5
 Git commit:        e91ed5707e
 Built:             Mon Dec 13 22:31:40 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

At the moment if you search for "facebook" you will get results about facebook, whereas you should probably get facebook.com/. We should prioritise such root URLs if they exist.

In Step 14/15 copying the data folder is failing, because the folder is not present. If I manually create the data folder, the image will build, but I can not start the container: `Traceback (most recent call last):

File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/venv/lib/python3.9/site-packages/tinysearchengine/app.py", line 13, in tiny_index = TinyIndex(Document, index_path, NUM_PAGES, PAGE_SIZE) File "/venv/lib/python3.9/site-packages/tinysearchengine/indexer.py", line 76, in init self.index_file = open(self.index_path, 'rb') FileNotFoundError: [Errno 2] No such file or directory: '/data/index.tinysearch'`

Which files have to be present in this data folder?

can you comment why pip uninstall need to be invoked three times in a row

https://github.com/mwmbl/mwmbl/blob/03ca368b2acb1a23edb839df7884452d7c26f81d/indexer/bootstrap.sh#L5

please

I have checked that some websites with different languages availability show the content in the language how was indexed.

I think that this content summary should be a registry based on the language of the web browser using it, and maybe a same website should be indexed several times to get content summary in different languages or made through the extension with prefixed languages.

In this picture, it is showed the http link in a Cyrillic-based language instead of English, that is being used by my web browser.

In this other picture, one of the links is showed in french.

While you typing a query the search results are updating and URL with this query also, there is no need to store intermediary URL queries in browser history, because it is bulking and make history less readable.

At the moment, the number of pages and the page size are stored in the code. This doesn't make sense as different indexes can have different page sizes and number of pages. Instead I suggest storing this metadata either:

1. in the index itself, in which case we could sacrifice the first 4096 bytes for metadata so as to maintain the physical memory page boundaries
2. or, we could use a separate file stored along with the index.

My current preference is for 1.

We can use the Bing API to:

1. Identify common search queries using the Autosuggest API. To do this we can query e.g. "n" to autosuggest and get back common queries that begin with "n", e.g. ["next", "news", ...]. This can then be bootstrapped by putting in these common queries to get longer queries. So, send "news" to autosuggest to get ["news uk", "news bbc", "news today"].
2. Given each query, retrieve the top N results for each query from the Web Search API.

We should ideally collect a dataset at least 2,000 queries, which we can split into development set and test set of 1,000 each.

For the evaluation we will want to filter the retrieved results to the same set of domains that we are currently restricted to (top HN scoring domains).

Merge request includes

• renamed package to mwmbl in pyproject.toml
• tinysearchengine and indexer modules have been moved into mwmbl package folder
• analyse module has been left as is in the root of the repo
• import statements in tinysearchengine now use mwmbl.tinysearchengine
• import statements in indexer now use mwmbl.indexer or mwmbl.tinysearchengine or relative imports like .paths
• import statements in analyse now use mwmbl.indexer or mwmbl.tinysearchengine
• final CMD in Dockerfile now uses updated path mwmbl.tinysearchengine.app
• fixed a couple of import statement errors in tinysearchengine/indexer.py

Notes for reviewer

• Fixes #15
• Its recommended to first merge #13 before this PR. However this PR is self sufficient but will have incomplete dependencies.
• Tested that the building the Dockerfile still works but does not run with the known problem that the index file is missing.
• Tested that the mwmbl.indexer.extract_local.py runs without failing imports (ends without doing anything since input_queue is empty)
• Unable to test mwmbl.tinysearchengine.app due to missing index file.

Reading the article over at https://daoudclarke.net/search%20engines/2022/07/10/non-profit-search-engine was interesting, goals like these are indeed worth fighting for:

Even if I make the web better for one person, it’s worth it. Because the way things are is just wrong.

Though, it is a bit puzzling that the choice in tooling does not follow that same spirit. Why is a project like this hosted on the Google of Git hosting services: GitHub?

Please consider moving to more open alternatives, e.g. GitLab.

• https://www.theregister.com/2022/06/30/software_freedom_conservancy_quits_github/

Something between lemmy and an indexer/search engine. Where the visibility for each website would be different in each instance based on the user ratings. Some instance could also block some domains. For example an instance for FOSS enthusiasts that blocked big tech domains. Or an instance for leftists that blocked western media.

To search a niche topic you could select (if it existed) the instance focusing on that. Instead of doing kung-fu on a general search engine.

It would show all links, except the ones for the blocked domains, instead of only the ones posted by the users, as lemmy does. But instead of only the links you would also have a button to show a comment section to have a community aspect like lemmy. Comments would have multiple levels, votes, and different sorting options just like lemmy.

The rating system would require community moderation. A pyramidal trust-based moderation system like discourse so that an instance admin would only have to deal with a few users to make sure the instance remained free of bots and ill-intentioned users skewing the ratings.

During development of crawler code, it would be significantly helpful to have a crawler API endpoint that let crawlers access certain testing pages.

Some ideas:

• api.crawler-test.mwmbl.org: The API endpoint.
• target.crawler-test.mwmbl.org/200.html, target.crawler-test.mwmbl.org/404.html, target.crawler-test.mwmbl.org/noindex.html, target.crawler-test.mwmbl.org/disallow.html, ...: Test cases.
• disallow-all.crawler-test.mwmbl.org: Another test case.
• etc.

Launching such API endpoint locally seems useful too. In principle we can share most of the code between public test endpoint and custom local test endpoint.

A request for the following sites to be added to the crawler

• codeberg
• lemmy.ml
• blue flashcards
• odysee
• kutt it
• ocean of pdf

Original request: https://github.com/mwmbl/crawler-extension/issues/34

Great projekt! Searching for e.g. wikipedia, new york times or hacker news does not lead to the start page of the website but to random sub pages. Same issue when searching for the exact start page url directly.