Merge pull request #177 from topkecleon/develop

Prepare 1.45 release with final webhook and get-file support

Author: gnadelwartz

README.html

@@ -145,6 +145,7 @@
     pre > code.sourceCode { white-space: pre; position: relative; }
     pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
     pre > code.sourceCode > span:empty { height: 1.2em; }
+    .sourceCode { overflow: visible; }
     code.sourceCode > span { color: inherit; text-decoration: inherit; }
     div.sourceCode { margin: 1em 0; }
     pre.sourceCode { margin: 0; }
@@ -341,7 +342,7 @@ <h2>Security Considerations</h2>
 <p>Running a Telegram Bot means it is connected to the public and you never know what's send to your Bot.</p>
 <p>Bash scripts in general are not designed to be bulletproof, so consider this Bot as a proof of concept. Bash programmers often struggle with 'quoting hell' and globbing, see <a href="https://unix.stackexchange.com/questions/171346/security-implications-of-forgetting-to-quote-a-variable-in-bash-posix-shells">Implications of wrong quoting</a>.</p>
 <p>Whenever you are processing input from untrusted sources (messages, files, network) you must be as careful as possible (e.g. set IFS appropriately, disable globbing with <code>set -f</code> and quote everything). In addition remove unused scripts and examples from your Bot (e.g. everything in <code>example/</code>) and disable/remove all unused bot commands.</p>
-<p>It's important to escape or remove <code>$</code> in input from user, files or network (<em>as bashbot does</em>). One of the powerful features of Unix shells is variable and command substitution using <code>${}</code> and <code>$()</code> can lead to remote code execution (RCE) or remote information disclosure (RID) bugs if unescaped <code>$</code> is included in untrusted input (e.g. <code>$$</code> or <code>$(rm -rf /*)</code>).</p>
+<p>It's important to escape or remove <code>$</code> and ` in input from user, files or network (<em>as bashbot does</em>). One of the powerful features of Unix shells is variable and command substitution using <code>${var}</code>, <code>$(cmd)</code> and `cmd` can lead to remote code execution (RCE) or remote information disclosure (RID) bugs if unescaped <code>$</code> or ` is included in untrusted input (e.g. <code>$$</code> or <code>$(rm -rf /*)</code>).</p>
 <p>A powerful tool to improve your scripts is <code>shellcheck</code>. You can <a href="https://www.shellcheck.net/">use it online</a> or <a href="https://github.com/koalaman/shellcheck#installing">install shellcheck locally</a>. Shellcheck is used extensively in bashbot development to ensure a high code quality (e.g. it's not allowed to push changes without passing all shellcheck tests). In addition bashbot has a <a href="doc/7_develop.md">test suite</a> to check if important functionality is working as expected.</p>
 <h3>Use printf whenever possible</h3>
 <p>If you're writing a script that accepts external input (e.g. from the user as arguments or the file system), you shouldn't use echo to display it. <a href="https://unix.stackexchange.com/a/6581">Use printf whenever possible</a>.</p>
@@ -392,6 +393,6 @@ <h3>Blocked by telegram?</h3>
 <p>@Gnadelwartz</p>
 <h2>That's it all guys!</h2>
 <p>If you feel that there's something missing or if you found a bug, feel free to submit a pull request!</p>
-<h4>$$VERSION$$ v1.41-0-gad1b91f</h4>
+<h4>$$VERSION$$ v1.5-0-g8adca9b</h4>
 </body>
 </html>

README.md

@@ -146,8 +146,9 @@ Whenever you are processing input from untrusted sources (messages, files, netwo
 (e.g. set IFS appropriately, disable globbing with `set -f` and quote everything). In addition remove unused scripts and examples
 from your Bot (e.g. everything in `example/`) and disable/remove all unused bot commands.
 
-It's important to escape or remove `$` in input from user, files or network (_as bashbot does_).
-One of the powerful features of Unix shells is variable and command substitution using `${}` and `$()` can lead to remote code execution (RCE) or remote information disclosure (RID) bugs if unescaped `$` is included in untrusted input (e.g. `$$` or `$(rm -rf /*)`).
+It's important to escape or remove `$` and \` in input from user, files or network (_as bashbot does_).
+One of the powerful features of Unix shells is variable and command substitution using `${var}`, `$(cmd)` and \`cmd\` can lead to remote
+code execution (RCE) or remote information disclosure (RID) bugs if unescaped `$` or \` is included in untrusted input (e.g. `$$` or `$(rm -rf /*)`).
 
 A powerful tool to improve your scripts is `shellcheck`. You can [use it online](https://www.shellcheck.net/) or
 [install shellcheck locally](https://github.com/koalaman/shellcheck#installing). Shellcheck is used extensively in bashbot development
@@ -241,4 +242,4 @@ See `mycommnds.sh.dist` for an example.
 
 If you feel that there's something missing or if you found a bug, feel free to submit a pull request!
 
-#### $$VERSION$$ v1.41-0-gad1b91f
+#### $$VERSION$$ v1.5-0-g8adca9b

README.txt

@@ -200,10 +200,11 @@ Whenever you are processing input from untrusted sources (messages, files, netwo
 must be as careful as possible (e.g. set IFS appropriately, disable globbing with set -
 f and quote everything). In addition remove unused scripts and examples from your Bot
 (e.g. everything in example/) and disable/remove all unused bot commands.
-It's important to escape or remove $ in input from user, files or network (as bashbot
-does). One of the powerful features of Unix shells is variable and command substitution
-using ${} and $() can lead to remote code execution (RCE) or remote information disclosure
-(RID) bugs if unescaped $ is included in untrusted input (e.g. $$ or $(rm -rf /*)).
+It's important to escape or remove $ and ` in input from user, files or network (as
+bashbot does). One of the powerful features of Unix shells is variable and command
+substitution using ${var}, $(cmd) and `cmd` can lead to remote code execution (RCE) or
+remote information disclosure (RID) bugs if unescaped $ or ` is included in untrusted
+input (e.g. $$ or $(rm -rf /*)).
 A powerful tool to improve your scripts is shellcheck. You can use it online [https://
 www.shellcheck.net/] or install shellcheck locally [https://github.com/koalaman/
 shellcheck#installing]. Shellcheck is used extensively in bashbot development to ensure a
@@ -318,5 +319,5 @@ That's it all guys!
 If you feel that there's something missing or if you found a bug, feel free to submit a
 pull request!
 
-$$VERSION$$ v1.41-0-gad1b91f
+$$VERSION$$ v1.5-0-g8adca9b
 

addons/antiFlood.sh

@@ -4,7 +4,7 @@
 # this addon counts how many files, e.g. stickers, are sent to
 # a chat and takes actions if threshold is reached
 #  
-#### $$VERSION$$ v1.40-0-gf9dab50
+#### $$VERSION$$ v1.5-0-g8adca9b
 
 # used events:
 #

addons/example.sh

@@ -4,7 +4,7 @@
 # Addons can register to bashbot events at startup
 # by providing their name and a callback per event
 #
-#### $$VERSION$$ v1.40-0-gf9dab50
+#### $$VERSION$$ v1.5-0-g8adca9b
 #
 # If an event occurs each registered event function is called.
 #

bashbot.rc

@@ -5,9 +5,10 @@
 #
 # tested on: ubuntu, opensuse, debian
 #
-#### $$VERSION$$ v1.40-0-gf9dab50
+#### $$VERSION$$ v1.5-0-g8adca9b
 # shellcheck disable=SC2009
 # shellcheck disable=SC2181
+# shellcheck disable=SC2250
 
 #
 ### BEGIN INIT INFO
@@ -31,13 +32,17 @@ runas="nobody"
 
 # uncomment one of the example lines to fit your system 
 # runcmd="su ${runas} -s /bin/bash -c "      # runasuser with *su*
-# runcmd="runuser ${runas} -s /bin/bash -c " # runasuser with *runuser*
+# runcmd="/usr/sbin/runuser ${runas} -s /bin/bash -c " # runasuser with *runuser*
 
 # edit the values of the following lines to fit your config:
-# your bot installation dir
-bashbot="cd /usr/local/telegram-bot-bash; /usr/local/telegram-bot-bash/bashbot.sh"
 # your bot name as given to botfather, e.g. mysomething_bot
 name=""
+# your bot installation dir
+bashbotdir="/usr/local/telegram-bot-bash"
+databotdir="${bashbotdir}/data-bot-bash"
+# programs to run
+bashbot="cd ${bashbotdir}; ${bashbotdir}/bashbot.sh"
+webhook="cd ${bashbotdir}; nohup ${bashbotdir}/bin/process_batch.sh --startbot --watch ${databotdir}/webhook-fifo-${name}"
 # set additionl parameter, e.g. debug
 mode=""
 
@@ -48,29 +53,54 @@ mode=""
 
 case "$1" in
 'start')
-	# shellcheck disable=SC2250
 	$runcmd "$bashbot start $mode" # >/dev/null 2>&1 </dev/null
 	RETVAL=$?
 	;;
+'starthook')
+	printf "Starting bashbot in webhook mode ... "
+	$runcmd "$webhook $mode </dev/null &>>${bashbotdir}/logs/WEBHOOK.log &" # >/dev/null 2>&1 </dev/null
+	sleep 1
+	$0 status
+	RETVAL=$?
+	;;
 'stop')
-	# shellcheck disable=SC2250
 	$runcmd "$bashbot stop $mode"
 	RETVAL=$?
 	;;
+'stophook')
+	printf "Stopping bashbot webhook mode ... "
+	KILLID="$(ps -f -u "${runas}" | grep "process_batch.sh --startbot" | sed -E 's/[^0-9]+([0-9]+).*/\1/')"
+	if [ -n "${KILLID}" ]; then
+		$runcmd "kill $(printf "%s" "${KILLID}" | tr -s "\r\n" "  " )"
+		sleep 1
+		$0 status
+	fi
+	RETVAL=$?
+	;;
 'status')
 	ps -f -u "${runas}" | grep "${name}" | grep -qF "bashbot.sh startbot"
 	if [ "$?" = "0" ]; then
-		printf "bashbot (%s) is running\n" "${name}"
+		printf "bashbot (%s) is running in poll mode\n" "${name}"
 		RETVAL=0
 	else
-		printf "bashbot (%s) is stopped\n" "${name}"
-		RETVAL=1
+		ps -f -u "${runas}" | grep "${name}" | grep -qF "process_batch.sh --startbot"
+		if [ "$?" = "0" ]; then
+			printf "bashbot (%s) is running in webhook mode\n" "${name}"
+			RETVAL=0
+		else
+			printf "bashbot (%s) is stopped\n" "${name}"
+			RETVAL=1
+		fi
 	fi
 	;;
 'restart'|'reload')
 	$0 stop; $0 start 
 	RETVAL=$?
 	;;
+'restarthook'|'reloadhook')
+	$0 stophook; $0 starthook 
+	RETVAL=$?
+	;;
 'restartback')
 	$0 suspendback; $0 resumeback 
 	RETVAL=$?
@@ -86,7 +116,8 @@ case "$1" in
 	fi  
 	;;
 *)
-	printf "%s\n" "Usage: $0 { start | stop | restart | reload | restartback | suspendback | resumeback | killback }"
+	printf "%s\n" "Usage: $0 [ start  | stop | restart | starthook |  stophook  | restarthook ]"
+	printf "%s\n" "       $0 [ status | restartback | suspendback  | resumeback | killback ]"
 	RETVAL=1
 	;;
 esac

bashbot.sh

@@ -30,7 +30,7 @@ BOTCOMMANDS="-h  help  init  start  stop  status  suspendback  resumeback  killb
 #     8 - curl/wget missing
 #     10 - not bash!
 #
-#### $$VERSION$$ v1.45-dev-26-g82a57a7
+#### $$VERSION$$ v1.5-0-g8adca9b
 ##################################################################
 
 # are we running in a terminal?
@@ -106,7 +106,7 @@ JsonEscape(){
 }
 # clean \ from escaped json string
 # $1 string, output cleaned string
-cleanEscaped(){	# remove "	all \ but  \n\u		\n or \r
+cleanEscape(){	# remove "	all \ but  \n\u		\n or \r
 	sed -E -e 's/\\"/+/g' -e 's/\\([^nu])/\1/g' -e 's/(\r|\n)//g' <<<"$1"
 }
 # check if $1 seems a valid token
@@ -358,7 +358,7 @@ declare -rx BOTTOKEN URL ME_URL
 declare -ax CMD
 declare -Ax UPD BOTSENT USER MESSAGE URLS CONTACT LOCATION CHAT FORWARD REPLYTO VENUE iQUERY iBUTTON
 declare -Ax SERVICE NEWMEMBER LEFTMEMBER PINNED MIGRATE
-export res CAPTION ME
+export res CAPTION ME BOTADMIN
 
 
 ###############
@@ -491,9 +491,8 @@ sendJson(){
 	# compose final json
 	json='{'"${chat} $(iconv -f utf-8 -t utf-8 -c <<<"$2")"'}'
 	if [ -n "${BASHBOTDEBUG}" ] ; then
-		log_update "sendJson (${DETECTED_CURL}) CHAT=${chat#*:} JSON=${2:0:100} URL=${3##*/}"
-		#									mask " and \ , remove newline from json
-		log_message "DEBUG sendJson ==========\n$("${JSONSHFILE}" -b -n  <<<"$(cleanEscaped "${json}")" 2>&1)"
+		log_update "sendJson (${DETECTED_CURL}) CHAT=${chat#*:} JSON=$(cleanEscape "${json:0:100}") URL=${3##*/}"
+		log_message "DEBUG sendJson ==========\n$("${JSONSHFILE}" -b -n  <<<"$(cleanEscape "${json}")" 2>&1)"
 	fi
 	# chat id not a number
 	if [[ "${chat}" == *"NAN\"," ]]; then
@@ -515,8 +514,8 @@ UPLOADDIR="${BASHBOT_UPLOAD:-${DATADIR}/upload}"
 # return final file name or empty string on error
 checkUploadFile() {
 	local err file="$2"
-	[[ "${file}" = *'..'* || "${file}" = '.'* ]] && err=1 	# no directory traversal
-	if [[ "${file}" = '/'* ]] ; then
+	[[ "${file}" == *'..'* || "${file}" == '.'* ]] && err=1 	# no directory traversal
+	if [[ "${file}" == '/'* ]] ; then
 		[[ ! "${file}" =~ ${FILE_REGEX} ]] && err=2	# absolute must match REGEX
 	else
 		file="${UPLOADDIR:-NOUPLOADDIR}/${file}"	# others must be in UPLOADDIR
@@ -537,6 +536,7 @@ checkUploadFile() {
 	    [ -n "${BASHBOTDEBUG}" ] && log_debug "$3: CHAT=$1 FILE=$2 MSG=${BOTSENT[DESCRIPTION]}"
 	    return 1
 	fi
+	printf "%s\n" "${file}"
 }
 
 
@@ -747,6 +747,16 @@ event_send() {
 	done
 }
 
+# cleanup activities on startup, called from startbot and resume background jobs
+# $1 action, timestamp for action is saved in config
+bot_cleanup() {
+	# cleanup countfile on startup
+	jssh_deleteKeyDB "CLEAN_COUNTER_DATABASE_ON_STARTUP" "${COUNTFILE}"
+        [ -f "${COUNTFILE}.jssh.flock" ] && rm -f "${COUNTFILE}.jssh.flock"
+	# store action time and cleanup botconfig on startup
+	[ -n "$1" ] && jssh_updateKeyDB "$1" "$(_date)" "${BOTCONFIG}"
+        [ -f "${BOTCONFIG}.jssh.flock" ] && rm -f "${BOTCONFIG}.jssh.flock"
+}
 
 # fallback version, full version is in  bin/bashbot_init.in.sh
 # initialize bot environment, user and permissions
@@ -788,6 +798,8 @@ fi
 # source the script with source as param to use functions in other scripts
 # do not execute if read from other scripts
 
+BOTADMIN="$(getConfigKey "botadmin")"
+
 if [ -z "${SOURCE}" ]; then
   ##############
   # internal options only for use from bashbot and developers
@@ -829,7 +841,8 @@ if [ -z "${SOURCE}" ]; then
 		;;
 	# finally starts the read update loop, internal use only
 	"startbot" )
-		_exec_if_function start_bot "$2"
+		_exec_if_function start_bot "$2" "polling mode"
+		_exec_if_function get_updates "$2"
 		debug_checks "end startbot" "$@"
 		exit
 		;;
@@ -899,7 +912,7 @@ if [ -z "${SOURCE}" ]; then
 			# shellcheck disable=SC2086
 			if kill ${BOTPID}; then
 				# inform botadmin about stop
-				send_normal_message "$(getConfigKey "botadmin")" "Bot ${ME} stopped ..." &
+				send_normal_message "${BOTADMIN}" "Bot ${ME} polling mode stopped ..." &
 				printf "${GREEN}OK. Bot stopped successfully.${NN}"
 			else
 				printf "${RED}An error occurred while stopping bot.${NN}"
@@ -912,7 +925,7 @@ if [ -z "${SOURCE}" ]; then
 		exit
 		;;
 	# suspend, resume or kill background jobs
-	"suspendb"*|"resumeb"*|"killb"*)
+	"suspendb"*|"resumeb"*|'restartb'*|"killb"*)
   		_is_function job_control || { printf "${RED}Module background is not available!${NN}"; exit 3; }
 		ME="$(getConfigKey "botname")"
 		job_control "$1"

bin/any_command.sh

@@ -21,7 +21,7 @@ USAGE='any_command.sh [-h|--help] [--force|--reference] bot_command args ...'
 #        AUTHOR: KayM (gnadelwartz), [email protected]
 #       CREATED: 30.01.2021 10:24
 #
-#### $$VERSION$$ v1.45-dev-7-ga9ed559
+#### $$VERSION$$ v1.5-0-g8adca9b
 #===============================================================================
 
 ####
@@ -68,7 +68,7 @@ fi
 # ready, do stuff here -----
 COMMAND="$1"
 if [ "$2" == "BOTADMIN" ]; then
-	ARG1="${BOT_ADMIN}"
+	ARG1="${BOTADMIN}"
 else
 	ARG1="$2"
 fi

bin/bashbot_env.inc.sh

@@ -13,16 +13,16 @@
 #        AUTHOR: KayM (gnadelwartz), [email protected]
 #       CREATED: 18.12.2020 12:27
 #
-#### $$VERSION$$ v1.40-0-gf9dab50
+#### $$VERSION$$ v1.5-0-g8adca9b
 #===============================================================================
 
 ############
 # set where your bashbot lives
 export BASHBOT_HOME BASHBOT_ETC BASHBOT_VAR FILE_REGEX ME
 
 # default: one dir up 
-BASHBOT_HOME="$(cd "${BASH_SOURCE[0]%/*}" >/dev/null 2>&1 && pwd)/../"
-[ "${BASHBOT_HOME}" = "/../" ] && BASHBOT_HOME="../"
+BASHBOT_HOME="$(cd "${BASH_SOURCE[0]%/*}/../" >/dev/null 2>&1 && pwd)"
+[ "${BASHBOT_HOME}" = "" ] && BASHBOT_HOME="../"
 
 # set you own BASHBOT_HOME if different, e.g.
 # BASHBOT_HOME="/usr/local/telegram-bot-bash"
@@ -59,11 +59,14 @@ UPLOADDIR="${BASHBOT_VAR%/bin*}"
 FILE_REGEX="${UPLOADDIR}/.*"
 
 # get and check ADMIN and NAME
-BOT_ADMIN="$(getConfigKey "botadmin")"
-BOT_NAME="$(getConfigKey "botname")"
-ME="${BOT_NAME}"
-[[ -z "${BOT_ADMIN}" || "${BOT_ADMIN}" == "?" ]] && printf "%s\n" "${ORANGE}Warning: Botadmin not set, send bot command${NC} /start"
-[[ -z "${BOT_NAME}"  ]] && printf "%s\n" "${ORANGE}Warning: Botname not set, run bashbot.sh botname"
+BOTNAME="$(getConfigKey "botname")"
+ME="${BOTNAME}"
+[[ -z "${BOTADMIN}" || "${BOTADMIN}" == "?" ]] && printf "%s\n" "${ORANGE}Warning: Botadmin not set, send bot command${NC} /start"
+[[ -z "${BOTNAME}"  ]] && printf "%s\n" "${ORANGE}Warning: Botname not set, run bashbot.sh botname"
+
+# default webhook pipe
+export WEBHOOK="${DATADIR}/webhook-fifo-${ME}"
+
 
 # output command result or Telegram response
 print_result() { jssh_printDB "BOTSENT" | sort -r; }

bin/bashbot_init.inc.sh

@@ -11,7 +11,7 @@
 #        AUTHOR: KayM (gnadelwartz), [email protected]
 #       CREATED: 27.01.2021 13:42
 #
-#### $$VERSION$$ v1.45-dev-3-g429c230
+#### $$VERSION$$ v1.5-0-g8adca9b
 #===============================================================================
 # shellcheck disable=SC2059
 
@@ -76,7 +76,7 @@ bot_init() {
 		[ -n "${INTERACTIVE}" ] && read -r runuser
 	fi
 	# check if mycommands exist
-	if [ ! -r "${BASHBOT_ETC:-.}/mycommands.sh" ]; then
+	if [[ ! -r "${BASHBOT_ETC:-.}/mycommands.sh" && -r ${BASHBOT_ETC:-.}/mycommands.sh.dist ]]; then
 		printf "Mycommands.sh not found, copy ${GREY}<C>lean file, <E>xamples or <N>one${NC} to mycommands.sh? (c/e/N) N\b"
 		read -r ANSWER
 		[[ "${ANSWER}" =~ ^[cC] ]] && cp -f "${BASHBOT_ETC:-.}/mycommands.sh.clean" "${BASHBOT_ETC:-.}/mycommands.sh"
@@ -116,7 +116,7 @@ bot_init() {
 	if [ -w "bashbot.rc" ]; then
 		printf "Adjust user and botname in bashbot.rc ...\n"
 		sed -i '/^[# ]*runas=/ s|runas=.*$|runas="'"${touser}"'"|' "bashbot.rc"
-		sed -i '/^[# ]*bashbot=/ s|bashbot=.*$|bashbot="cd '"${PWD}"'; '"${PWD}"'/'"${0##*/}"'"|' "bashbot.rc"
+		sed -i '/^[# ]*bashbotdir=/ s|bashbotdir=.*$|bashbotdir="'"${PWD}"'"|' "bashbot.rc"
 		botname="$(getConfigKey "botname")"
 		[ -n "${botname}" ] && sed -i '/^[# ]*name=/ s|name=.*$|name="'"${botname}"'"|' "bashbot.rc"
 		printf "Done.\n"