Can't create job using Docker

[ScottCov]

I an installing on debian(12.10) with ES (8.17.4). When I run for the first time:

docker run -it --rm \
     -v ~/.fscrawler:/root/.fscrawler \
     -v ~/tmp:/tmp/es:ro \
     dadoonet/fscrawler job_name

I get the following and it doesn't ask me if I want to create the job:

00:15:42,467 WARN  [o.p.AbstractPluginManager] No 'plugins' root
00:15:42,538 INFO  [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [227.2mb/3.8gb=5.73%], RAM [1.9gb/15.4gb=12.51%], Swap [976.9mb/976.9mb=100.0%].
00:15:42,582 WARN  [f.p.e.c.f.s.FsSettingsLoader] Can not read settings from [/root/.fscrawler/job_name] with either /_settings.yaml, /_settings.json, /_settings/*. Falling back to default settings.
00:15:43,220 INFO  [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
00:15:43,221 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
00:15:44,135 WARN  [f.p.e.c.f.c.ElasticsearchClient] Failed to create elasticsearch client on Elasticsearch{nodes=[https://127.0.0.1:9200], index='fscrawler', indexFolder='fscrawler_folder', bulkSize=100, flushInterval=5s, byteSize=10mb, apiKey='', username='null', pipeline='null', pathPrefix='null', sslVerification=true, caCertificate='null', pushTemplates=true}. Message: Can not execute GET https://127.0.0.1:9200/ : Connection refused.
00:15:44,136 FATAL [f.p.e.c.f.c.FsCrawlerCli] We can not start Elasticsearch Client. Exiting.
fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClientException: Can not execute GET https://127.0.0.1:9200/ : Connection refused
        at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpCall(ElasticsearchClient.java:1046) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpGet(ElasticsearchClient.java:991) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]

I installed java and checked JAVA_HOME and get /usr/lib/jvm/java-17-openjdk-amd64 and here is my docker info:

docker info
Client: Docker Engine - Community
 Version:    28.1.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.35.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 28.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-33-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 15.49GiB
 Name: server1
 ID: 8fd35d9c-7757-4f35-b31e-6d4b286e1edf
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

Anyone have any idea what I am doing wrong?

[dadoonet]

It looks like a bug. Let me check later today. 😉

[dadoonet]

Could you run it in TRACE mode?

docker run -it --rm \
     -v ~/.fscrawler:/root/.fscrawler \
     -v ~/tmp:/tmp/es:ro \
     -v ~/logs:/root/logs \
     -e FS_JAVA_OPTS="-DLOG_LEVEL=trace" \
     dadoonet/fscrawler job_name

[ScottCov]

sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; disabled; preset: enabled)
     Active: active (running) since Tue 2025-04-22 07:12:23 EDT; 5min ago
       Docs: https://www.elastic.co
   Main PID: 184312 (java)
      Tasks: 81 (limit: 18957)
     Memory: 8.4G
        CPU: 1min 16.484s
     CGroup: /system.slice/elasticsearch.service
             ├─184312 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.h>
             ├─184452 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 >
             └─184473 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Apr 22 07:11:37 server1 systemd[1]: Starting elasticsearch.service - Elasticsearch...
Apr 22 07:11:44 server1 systemd-entrypoint[184452]: CompileCommand: dontinline java/lang/invoke/MethodHandle.setAsTypeCache bool dontinline = true
Apr 22 07:11:44 server1 systemd-entrypoint[184452]: CompileCommand: dontinline java/lang/invoke/MethodHandle.asTypeUncached bool dontinline = true
Apr 22 07:12:23 server1 systemd[1]: Started elasticsearch.service - Elasticsearch.

11:13:49,255 DEBUG [f.p.e.c.p.FsCrawlerPluginsManager] Loading plugins                                                                                                                                            
11:13:49,257 WARN  [o.p.AbstractPluginManager] No 'plugins' root                                                                                                                                                  
11:13:49,312 INFO  [f.p.e.c.f.c.BootstrapChecks] Memory [Free/Total=Percent]: HEAP [226.5mb/3.8gb=5.71%], RAM [5.3gb/15.4gb=34.25%], Swap [976.9mb/976.9mb=100.0%].                                               
11:13:49,313 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] Copying [6/_settings.json]...                                                                                                                                      
11:13:49,348 DEBUG [f.p.e.c.f.f.FsCrawlerUtil] Copying [6/_settings_folder.json]...                                                                                                                               
11:13:49,350 DEBUG [f.p.e.c.f.c.FsCrawlerCli] Starting job [job_name]...                                                                                                                                          
11:13:49,352 TRACE [f.p.e.c.f.s.FsSettingsLoader] Trying to read settings from [/root/.fscrawler/job_name/_settings.yaml] file                                                                                    
11:13:49,353 TRACE [f.p.e.c.f.s.FsSettingsLoader] Trying to read settings from [/root/.fscrawler/job_name/_settings.json] file                                                                                    
11:13:49,353 TRACE [f.p.e.c.f.s.FsSettingsLoader] Trying to read settings from [/root/.fscrawler/job_name/_settings] directory                                                                                    
11:13:49,353 WARN  [f.p.e.c.f.s.FsSettingsLoader] Can not read settings from [/root/.fscrawler/job_name] with either /_settings.yaml, /_settings.json, /_settings/*. Falling back to default settings.            
11:13:49,892 DEBUG [f.p.e.c.f.s.FsSettingsLoader] Successfully loaded settings from [fscrawler-default.properties] and []                                                                                         
11:13:49,970 TRACE [f.p.e.c.f.s.FsSettingsLoader] Loaded settings [FsSettings{name='fscrawler', fs=Fs{url='/tmp/es', updateRate=15m, includes=null, excludes=[*/~*], filters=null, jsonSupport=false, filenameAsId=false, addFilesize=true, removeDeleted=true, addAsInnerObject=false, storeSource=false, indexContent=true, indexedChars=null, attributesSupport=false, rawMetadata=false, xmlSupport=false, checksum='null', indexFolders=true, langDetect=false, continueOnError=false, ocr=Ocr{language='eng', path='null', dataPath='null', outputType='txt', enabled=true, pdfStrategy='ocr_and_text', pageSegMode='1', preserveInterwordSpacing='false'}, ignoreAbove=null, followSymlinks=false, tikaConfigPath='null'}, server=Server{hostname='null', port=0, username='null', protocol='local', pemPath='null'}, elasticsearch=Elasticsearch{nodes=[https://127.0.0.1:9200], index='fscrawler', indexFolder='fscrawler_folder', bulkSize=100, flushInterval=5s, byteSize=10mb, apiKey='', username='null', pipeline='null', pathPrefix='null', sslVerification=true, caCertificate='null', pushTemplates=true}, rest=Rest{url='http://127.0.0.1:8080/fscrawler', enableCors=false}, tags=Tags{metaFilename='.meta.yml'}}]
11:13:50,142 TRACE [f.p.e.c.f.c.FsCrawlerCli] settings used for this crawler: [---
name: "fscrawler"
fs:
  url: "/tmp/es"
  update_rate: "15m"
  excludes:
  - "*/~*"
  json_support: false
  add_as_inner_object: false
  xml_support: false
  follow_symlinks: false
  remove_deleted: true
  continue_on_error: false
  filename_as_id: false
  add_filesize: true
  attributes_support: false
  store_source: false
  index_content: true
  raw_metadata: false
  index_folders: true
  lang_detect: false
  ocr:
    enabled: true
    language: "eng"
    output_type: "txt"
    pdf_strategy: "ocr_and_text"
    page_seg_mode: 1
    preserve_interword_spacing: false
server:
  port: 0
  protocol: "local"
elasticsearch:
  nodes:
  - url: "https://127.0.0.1:9200"
  index: "fscrawler"
  index_folder: "fscrawler_folder"
  bulk_size: 100
  flush_interval: "5s"
  byte_size: "10mb"
  ssl_verification: true
  push_templates: true
  semantic_search: true
rest:
  url: "http://127.0.0.1:8080/fscrawler"
  enable_cors: false
tags:
  meta_filename: ".meta.yml"
]
11:13:50,157 DEBUG [f.p.e.c.p.FsCrawlerPluginsManager] Starting plugins
11:13:50,189 DEBUG [f.p.e.c.p.FsCrawlerPluginsManager] Found FsCrawlerExtensionFsProvider extension for type [http]
11:13:50,189 DEBUG [f.p.e.c.p.FsCrawlerPluginsManager] Found FsCrawlerExtensionFsProvider extension for type [local]
11:13:50,190 DEBUG [f.p.e.c.p.FsCrawlerPluginsManager] Found FsCrawlerExtensionFsProvider extension for type [s3]
11:13:50,208 DEBUG [f.p.e.c.f.FsParserAbstract] creating fs crawler thread [fscrawler] for [/tmp/es] every [15m]
11:13:50,210 DEBUG [f.p.e.c.f.FsParserAbstract] We are going to use [.meta.yml] as meta file if found whil crawling dirs
11:13:50,218 INFO  [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
11:13:50,220 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
11:13:50,320 DEBUG [f.p.e.c.f.c.ElasticsearchClient] get version
11:13:50,320 TRACE [f.p.e.c.f.c.ElasticsearchClient] Calling GET https://127.0.0.1:9200/ with params []
11:13:51,105 WARN  [f.p.e.c.f.c.ElasticsearchClient] Failed to create elasticsearch client on Elasticsearch{nodes=[https://127.0.0.1:9200], index='fscrawler', indexFolder='fscrawler_folder', bulkSize=100, flushInterval=5s, byteSize=10mb, apiKey='', username='null', pipeline='null', pathPrefix='null', sslVerification=true, caCertificate='null', pushTemplates=true}. Message: Can not execute GET https://127.0.0.1:9200/ : Connection refused.
11:13:51,105 FATAL [f.p.e.c.f.c.FsCrawlerCli] We can not start Elasticsearch Client. Exiting.
fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClientException: Can not execute GET https://127.0.0.1:9200/ : Connection refused
        at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpCall(ElasticsearchClient.java:1046) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpGet(ElasticsearchClient.java:991) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.getVersion(ElasticsearchClient.java:253) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.start(ElasticsearchClient.java:172) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.service.FsCrawlerManagementServiceElasticsearchImpl.start(FsCrawlerManagementServiceElasticsearchImpl.java:65) ~[fscrawler-core-2.10-SNAPSHOT.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl.start(FsCrawlerImpl.java:127) ~[fscrawler-core-2.10-SNAPSHOT.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.startEsClient(FsCrawlerCli.java:390) [fscrawler-cli-2.10-SNAPSHOT.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.runner(FsCrawlerCli.java:366) [fscrawler-cli-2.10-SNAPSHOT.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.cli.FsCrawlerCli.main(FsCrawlerCli.java:134) [fscrawler-cli-2.10-SNAPSHOT.jar:?]
Caused by: jakarta.ws.rs.ProcessingException: java.net.ConnectException: Connection refused
        at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:288) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:687) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:709) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:703) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-3.1.10.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-3.1.10.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-3.1.10.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:391) ~[jersey-common-3.1.10.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:703) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) ~[jersey-client-3.1.10.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpCall(ElasticsearchClient.java:1020) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
        ... 8 more
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.connect0(Native Method) ~[?:?]
        at java.base/sun.nio.ch.Net.connect(Net.java:589) ~[?:?]
        at java.base/sun.nio.ch.Net.connect(Net.java:578) ~[?:?]
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:583) ~[?:?]
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
        at java.base/java.net.Socket.connect(Socket.java:751) ~[?:?]
        at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304) ~[?:?]
        at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?]
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531) ~[?:?]
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636) ~[?:?]
        at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) ~[?:?]
        at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:377) ~[?:?]
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193) ~[?:?]
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1252) ~[?:?]
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1138) ~[?:?]
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179) ~[?:?]
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1690) ~[?:?]
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1614) ~[?:?]
        at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531) ~[?:?]
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307) ~[?:?]
        at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:453) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:286) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:687) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:709) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:703) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-3.1.10.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-3.1.10.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-3.1.10.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:391) ~[jersey-common-3.1.10.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:703) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-3.1.10.jar:?]
        at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) ~[jersey-client-3.1.10.jar:?]
        at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.httpCall(ElasticsearchClient.java:1020) ~[fscrawler-elasticsearch-client-2.10-SNAPSHOT.jar:?]
        ... 8 more
11:13:51,127 DEBUG [f.p.e.c.f.FsCrawlerImpl] Closing FS crawler [fscrawler]
11:13:51,127 TRACE [f.p.e.c.f.FsParserAbstract] Closing the parser FsParserLocal
11:13:51,128 DEBUG [f.p.e.c.f.FsCrawlerImpl] FS crawler thread is now stopped
11:13:51,128 DEBUG [f.p.e.c.f.c.ElasticsearchClient] Closing Elasticsearch client manager
11:13:51,135 DEBUG [f.p.e.c.f.s.FsCrawlerManagementServiceElasticsearchImpl] Elasticsearch Management Service stopped
11:13:51,136 DEBUG [f.p.e.c.f.c.ElasticsearchClient] Closing Elasticsearch client manager
11:13:51,136 DEBUG [f.p.e.c.f.s.FsCrawlerDocumentServiceElasticsearchImpl] Elasticsearch Document Service stopped
11:13:51,137 DEBUG [f.p.e.c.f.FsCrawlerImpl] ES Client Manager stopped
11:13:51,138 INFO  [f.p.e.c.f.FsCrawlerImpl] FS crawler [fscrawler] stopped

[dadoonet]

Yes. Not recommended but you can still use:

mkdir -p ~/.fscrawler/job_name/_settings
cat <<EOM >~/.fscrawler/job_name/_settings/elasticsearch.yml
elasticsearch:
  nodes:
    - url: "https://127.0.0.1:9200"
  username: "elastic"
  password: "changeme"
  # ssl_verification: false # You might have to uncomment this one
EOM

[ScottCov]

Thanks David-I changed url since I think when using docker for 127.0.0.1 it will look in the local host of the container and it seems to be working but it doesn't stop and ask if I want to create the job which is fine but I am guessing I need to manually create _settings.yaml?

[dadoonet]

Correct. The file is now optional as everything can be set using env variables.
But I'm adding ATM a new setup option to create the sample file.

please note that a sample file is available from: https://github.com/dadoonet/fscrawler/blob/master/settings/src/main/resources/fr/pilato/elasticsearch/crawler/fs/settings/fscrawler-default.yaml

[dadoonet]

I created #2072 to fix hopefully those kind of issues. It's a new workflow but I think it's much better and opens doors for new possibilities.

[ScottCov]

Thanks David