Merge pull request #49 from okybaca/improvements

Improvements

Author: Orbiter

docs/contribute.md

@@ -1,36 +1,105 @@
-# How to contribute to source code and documentation?
+# How to contribute to YaCy
+
+
+## Be active in the forum
+
+[Community forum](https://community.searchlab.eu) keeps the community alive.  
+If you're advanced user, good place to start is helping the others in
+the forum and sharing your knowledge there.
+
+
+## Report a bug
+
+If you find a bug or you want to suggest an improvement, fill the 
+[github issue](https://github.com/yacy/yacy_search_server/issues).
+
+Please use factual and technical language and try to describe the bug in
+details. 
+
+Focus on what behavior you did expect and what YaCy did instead of it.
+
+See [the log](operation/logging.md) and try to find and attach the
+appropriate log lines. Increase the log
+[verbosity](operation/logging.md#verbosity), if neccessary.
+
+Be sure, that the issue could be replicated (describe what to do to see
+the malfunction again). Problem happening just once, without the possiblity
+to make it happen again, could be hardly fixed.
+
+Pace of issue-fixing is currently quite low, so if you can, repair the
+function yourself and do a github pull-request (see below).
+
+
+## Help writting documentation
+
+YaCy has many more functions than described in the [documentation](docs.md)
+and is still heavily under-documented.  You can help others by writting about your
+favourite feature, updating the old pages, fixing installation guide for
+your platform, correcting the mistakes or just spell-checking.  See 
+[github repository of documentation](https://github.com/yacy/yacy_net_homepage), 
+[documentation issues](https://github.com/yacy/yacy_search_server/issues?q=state%3Aopen%20label%3ADocumentation) 
+and github guide bellow.
+
+
+## Help developing YaCy 
+
+If you're a Java wizard, you're most warmly welcomed to be part of the
+development!  
+
+Although YaCy is developed [since 2003](https://en.wikipedia.org/wiki/YaCy), and still maintained,
+progress is sometimes slow. Therefore chances for including your code to
+mainstream are very high.
+
+You can pick an [issue](https://github.com/yacy/yacy_search_server/issues) to solve.
+They're well tagged with [labels](https://github.com/yacy/yacy_search_server/labels) such as: 
+[good first issue](https://github.com/yacy/yacy_search_server/issues?q=state%3Aopen%20label%3A%22good%20first%20issue%22),
+[bug](https://github.com/yacy/yacy_search_server/issues?q=is%3Aissue%20state%3Aopen%20label%3Abug),
+[crawler](https://github.com/yacy/yacy_search_server/issues?q=state%3Aopen%20label%3Acrawler),
+[search](https://github.com/yacy/yacy_search_server/issues?q=state%3Aopen%20label%3Asearch),
+[index](https://github.com/yacy/yacy_search_server/issues?q=state%3Aopen%20label%3Aindex),
+[network](https://github.com/yacy/yacy_search_server/issues?q=state%3Aopen%20label%3Anetwork),
+[releasing](https://github.com/yacy/yacy_search_server/issues?q=state%3Aopen%20label%3Areleasing),
+[developer](https://github.com/yacy/yacy_search_server/issues?q=state%3Aopen%20label%3A%22developer%20issue%22),
+etc.
+ 
+You can improve what annoys you, personally, or craft a feature you like.
+
+Before adding a major feature, consult
+[@orbiter](https://github.com/Orbiter), the main developer, or 
+[the forum](https://community.searchlab.eu/).
+
+
+
+## Step by step guide for Github
 
 Basically, your contribution to the code and documentation is possible using
 github.com. Create account there, fork the official repository, clone it to
 your local machine, make a branch, modify files, commit changes to github. 
 Finally make a pull request, so your contribution could be merged into
 master branch.
 
-Step by step guide:
 
-(or follow the github contribution guide
-https://docs.github.com/en/get-started/quickstart/contributing-to-projects)
+(or follow the [github contribution guide](https://docs.github.com/en/get-started/quickstart/contributing-to-projects))
 
 
 
-## this is for the first time only
+### this is for the first time only
 
 * create a github.com account
 
 * log into github
 
-* fork repository https://github.com/yacy/yacy_search_server for a yacy
+* fork repository <https://github.com/yacy/yacy_search_server> for a yacy
   software modification, 
-  or https://github.com/yacy/yacy_net_homepage for documentation editing, 
+  or <https://github.com/yacy/yacy_net_homepage> for documentation editing, 
   respectively (use a 'fork' button on top right part of github.com)
 
 * your own fork is now at url:
   https://github.com/YOURUSERNAMEHERE/yacy_net_homepage
   or:
   https://github.com/YOURUSERNAMEHERE/yacy_search_server
 
-* set-up a ssh key using this guide:
-  https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account
+* set-up a ssh key using [this guide](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)
 
 * on your local machine, clone your new fork using commandline:
 ```
@@ -48,7 +117,7 @@ https://docs.github.com/en/get-started/quickstart/contributing-to-projects)
   git config --global user.name "YOURUSERNAMEHERE"
 ```
 
-## this is the regular workflow: 
+### for every next change
 
 * create a new branch under your working directory:
 ```
@@ -77,9 +146,9 @@ https://docs.github.com/en/get-started/quickstart/contributing-to-projects)
   to the master yacy branch. 
 
 * you can see all open pull request by you and other people at:
-  https://github.com/yacy/yacy_search_server/pulls
+  <https://github.com/yacy/yacy_search_server/pulls>
   or
-  https://github.com/yacy/yacy_net_homepage/pulls
+  <https://github.com/yacy/yacy_net_homepage/pulls>
 
 * wait until the mantainers merge your pull request
 

docs/docs.md

@@ -26,10 +26,11 @@
 
 
 ## Operation
-* [Index Creation - Crawl Start](operation/crawlstart_p.md)
+* [Advanced Crawler](operation/crawlstart_p.md)
 * [Setting the ranking rules](operation/ranking.md)
 * [YaCy config settings](operation/yacy_conf.md)
 * [RWI index distribution](operation/rwi-index-distribution.md)
+* [Yacy Packs](operation/yacy-pack.md)
 * [Autoupdate](operation/autoupdate.md)
 * [Portforwarding](operation/portforwarding.md)
 * [Using the YaCy Front-End over HTTPS](operation/yacyoverhttps.md)
@@ -52,7 +53,8 @@
 ## Converted from old-wiki
 may be outdated, you can help the community by checking and [improving](contribute.md) the pages
 
-...
+* [Index Export and Import](operation/index-export-import.md)
+* [Advanced Search Parameters](operation/search-parameters.md)
 
 ## Old and obsolete
 The original YaCy wiki is closed now (no new registration or editing) and

docs/download_installation.md

@@ -23,7 +23,7 @@ If you don't have Docker installed, get it from [https://docs.docker.com/get-doc
 
 
 * <a class="btn btn-success btn" href="https://download.yacy.net/yacy_v1.924_20201214_10042.exe" role="button">Download YaCy for Windows</a> from [https://download.yacy.net/yacy_v1.924_20201214_10042.exe](https://download.yacy.net/yacy_v1.924_20201214_10042.exe)
-* <a class="btn btn-success btn" href="https://download.yacy.net/yacy_v1.930_202405130205_59c0cb0f3.tar.gz" role="button">Download Yacy for Linux</a> from [https://download.yacy.net/yacy_v1.930_202405130205_59c0cb0f3.tar.gz](https://download.yacy.net/yacy_v1.930_202405130205_59c0cb0f3.tar.gz)
+* <a class="btn btn-success btn" href="https://download.yacy.net/yacy_v1.940_202405270005_70454654f.tar.gz" role="button">Download Yacy for Linux</a> from [https://download.yacy.net/yacy_v1.940_202405270005_70454654f.tar.gz](https://download.yacy.net/yacy_v1.940_202405270005_70454654f.tar.gz)
 * <a class="btn btn-success btn" href="https://download.yacy.net/yacy_v1.930_202405130205_59c0cb0f3.dmg" role="button">Download YaCy for macOS</a> from [https://download.yacy.net/yacy_v1.930_202405130205_59c0cb0f3.dmg](https://download.yacy.net/yacy_v1.930_202405130205_59c0cb0f3.dmg)
 * <a class="btn btn-success btn" href="https://release.yacy.net/yacy_latest.tar.gz" role="button">Download latest developer release</A> for Linux from [https://release.yacy.net/](https://release.yacy.net/)
 

docs/img/indexexpimp1.jpg

@@ -0,0 +1,49 @@
+# Index export and import
+
+Since the development version 1.83 build 9250 YaCy has now the long
+awaited feature to handle the index data in a more convenient way: An
+ex- and import feature has been implemented.
+
+It is no longer necessary to study complicated manuals for the merge of
+two solr indexes and/or setup an additional stand-alone instance for
+index merging tasks - thanks to the great work of Orbiter - YaCy
+delivers now a powerful ex- and import feature out-of-the-box\!
+
+## How to do that? 
+
+Here's a short tutorial:
+
+![Index Export screenshot](../img/indexexpimp1.jpg)]
+
+1\. On the machine you want to export the index data open a browser and
+navigate to <http://localhost:8090/IndexExport_p.html>
+
+Leave the settings as is, XML (Rich and full-text Solr data, one
+document per line in one large xml file, can be processed with shell
+tools, can be imported with `DATA/SURROGATE/in/`) because it's the best
+choice for the consistency of your data.
+
+2\. Press the 'Export URLs'-Button and grab some coffee :-)
+
+![Index Export screenshot](../img/indexexpimp2.jpg)
+
+3\. On the machine you want to import the index data, simply put the
+exported XML file into the following subdirectory of YaCy: `\DATA\SURROGATES\in`
+
+You can do this during YaCy is running - No need to shut it down first\!
+
+4\. Voilà - The import process starts automatically and is blazing fast
+- even on older machines. After the import process is completed, you can
+search trough the data instantly - reindexing is superfluous.
+
+
+
+
+
+_Converted from
+„<http://wiki.yacy.de/index.php?title=En:IndexExpImp&oldid=22656>“, may be
+outdated_
+
+
+
+

docs/img/indexexpimp2.jpg

@@ -0,0 +1,99 @@
+# Using Advanced Search Parameters
+
+Most search requests usually contain just one word. If more than one
+word is used, all words will be assumed to be part of an AND relation.
+Besides searching for certain words with YaCy, you can use more advanced
+methods to put a search request:
+
+## Excluding a Word
+
+To exclude a word from a search, a minus (`-`) can be used: for example,
+if searching for *jaguar* produces too many results associated with cars
+when in fact you are looking for the animal, searching for `jaguar -car`
+might lead to better results.
+
+## NEAR
+
+`NEAR` can be used to rank results higher if search words appear in the
+text close to each other. Example: `apache server NEAR`.
+
+ It does not matter where NEAR is located in the search term.  `apache
+server NEAR` and `apache NEAR server` should return the same results.
+
+## site:
+
+`findsomething site:yacy.net` will limit the results to the domain
+yacy.net, subdomains excluded. See `tld:` operator also.
+
+## tld:
+
+`findsomething tld:co.uk` will limit the results to domains ending with
+`*.co.uk.*` This can also be used to search on subdomains.
+
+## inurl:
+
+`findsomething inurl:source` will limit the results to URLs which
+contain the phrase "source".
+
+## filetype:
+
+`findsomething filetype:pdf` will limit the results to URLs which end
+with `.pdf`.
+
+## LANGUAGE:
+
+`findsomething LANGUAGE:en` will rank results in English language
+higher. (Note: Language detection is still very experimental\!)
+
+## RECENT
+
+`findsomething RECENT` will rank recently crawled pages higher.
+
+## Protocol
+
+`findsomething /ftp` will limit the results to URLs with FTP protocol
+(ftp://). List of available protocol: `/https` `/http` `/ftp` `/smb` or
+`/file`
+
+## author:
+
+`findsomething author:busch` will limit the results to URLs with author
+"busch". `findsomething author:(Wilhelm busch)` will limit the results
+to URLs with author "Wilhelm busch".
+
+## Date Search
+
+To search for results mentioning a specific date. 
+
+*Note: to support date search [solr index field](../dev/solr-schema.md#optional-but-recommended) `dates_in_content_dts` must be
+switched on*
+
+
+### on:
+
+`findsomething on:2016/01/01` will limit the results to URLs which
+contain the given date in the content.
+
+
+### from: to:
+
+`findsomething from:2016/01/01` will limit the results to URLs which
+contain a date on or after the `from:` parameter. 
+ 
+`findsomething to:2016/01/01` will limit the results to URLs which
+contain a date on or before the `to:` parameter.
+  
+Both can be combinded to limit results to the given date range
+`findsomething from:2016/01/01 to:2016/12/31`
+
+
+
+
+
+_Converted from
+<https://wiki.yacy.net/index.php?title=En:SearchParameters&oldid=22861> may
+be outdated_
+
+
+
+